You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2018/03/03 01:12:46 UTC

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 3 - Still Unstable

Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/3/

6 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage

Error Message:
SolrCore.getOpenCount()==2

Stack Trace:
java.lang.RuntimeException: SolrCore.getOpenCount()==2
	at __randomizedtesting.SeedInfo.seed([51AD4128F49876FC]:0)
	at org.apache.solr.util.TestHarness.close(TestHarness.java:379)
	at org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:794)
	at org.apache.solr.cloud.AbstractZkTestCase.azt_afterClass(AbstractZkTestCase.java:147)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:897)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.rest.TestManagedResourceStorage:     1) Thread[id=9124, name=ProcessThread(sid:0 cport:53267):, state=WAITING, group=TGRP-TestManagedResourceStorage]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:123)    2) Thread[id=9120, name=Thread-2462, state=WAITING, group=TGRP-TestManagedResourceStorage]         at java.lang.Object.wait(Native Method)         at java.lang.Thread.join(Thread.java:1252)         at java.lang.Thread.join(Thread.java:1326)         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:313)         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:310)         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:491)    3) Thread[id=9122, name=SessionTracker, state=TIMED_WAITING, group=TGRP-TestManagedResourceStorage]         at java.lang.Object.wait(Native Method)         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:147)    4) Thread[id=9121, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestManagedResourceStorage]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.rest.TestManagedResourceStorage: 
   1) Thread[id=9124, name=ProcessThread(sid:0 cport:53267):, state=WAITING, group=TGRP-TestManagedResourceStorage]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:123)
   2) Thread[id=9120, name=Thread-2462, state=WAITING, group=TGRP-TestManagedResourceStorage]
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1252)
        at java.lang.Thread.join(Thread.java:1326)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:313)
        at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:310)
        at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:491)
   3) Thread[id=9122, name=SessionTracker, state=TIMED_WAITING, group=TGRP-TestManagedResourceStorage]
        at java.lang.Object.wait(Native Method)
        at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:147)
   4) Thread[id=9121, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestManagedResourceStorage]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([51AD4128F49876FC]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=9121, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestManagedResourceStorage]         at sun.nio.ch.IOUtil.drain(Native Method)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:104)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=9121, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestManagedResourceStorage]
        at sun.nio.ch.IOUtil.drain(Native Method)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:104)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([51AD4128F49876FC]:0)


FAILED:  org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN

Error Message:
expected:<1.0> but was:<0.0>

Stack Trace:
java.lang.AssertionError: expected:<1.0> but was:<0.0>
	at __randomizedtesting.SeedInfo.seed([E7287D79FADCB7DA:16890F29CF677D48]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:443)
	at org.junit.Assert.assertEquals(Assert.java:512)
	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload

Error Message:
Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>

Stack Trace:
java.lang.AssertionError: Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>
	at __randomizedtesting.SeedInfo.seed([51AD4128F49876FC:53149A4487DBAA31]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload(TestJmxIntegration.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart

Error Message:
IOException occured when talking to server at: http://127.0.0.1:43903/solr/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:43903/solr/collection1
	at __randomizedtesting.SeedInfo.seed([51AD4128F49876FC:895A85CC5F43B4A0]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
	at org.apache.solr.handler.TestReplicationHandler.index(TestReplicationHandler.java:180)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart(TestReplicationHandler.java:643)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:43903 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 47 more




Build Log:
[...truncated 12256 lines...]
   [junit4] Suite: org.apache.solr.core.TestJmxIntegration
   [junit4]   2> 198809 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.core.TestJmxIntegration_51AD4128F49876FC-001/init-core-data-001
   [junit4]   2> 198810 WARN  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 198810 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 198812 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 198813 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 198814 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 198839 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 198863 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 198951 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 198993 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7e0fe570
   [junit4]   2> 199000 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7e0fe570
   [junit4]   2> 199000 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7e0fe570
   [junit4]   2> 199007 INFO  (coreLoadExecutor-1005-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 199008 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 199032 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 199049 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 199152 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 199159 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 199160 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7e0fe570
   [junit4]   2> 199160 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 199160 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.core.TestJmxIntegration_51AD4128F49876FC-001/init-core-data-001/]
   [junit4]   2> 199162 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=595609141, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 199209 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 199209 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 199210 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 199210 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 199211 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=774694259, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 199212 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@14214966[collection1] main]
   [junit4]   2> 199213 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 199220 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 199261 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 199261 INFO  (searcherExecutor-1006-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 199261 INFO  (searcherExecutor-1006-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 199261 INFO  (searcherExecutor-1006-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 199261 INFO  (searcherExecutor-1006-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 199261 INFO  (searcherExecutor-1006-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 199261 INFO  (searcherExecutor-1006-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 199262 INFO  (searcherExecutor-1006-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 199262 INFO  (searcherExecutor-1006-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 199262 INFO  (searcherExecutor-1006-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 199263 INFO  (searcherExecutor-1006-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@14214966[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 199263 INFO  (coreLoadExecutor-1005-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593873857865842688
   [junit4]   2> 199266 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 199268 INFO  (TEST-TestJmxIntegration.testJmxRegistration-seed#[51AD4128F49876FC]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testJmxRegistration
   [junit4]   2> 199389 INFO  (TEST-TestJmxIntegration.testJmxRegistration-seed#[51AD4128F49876FC]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testJmxRegistration
   [junit4]   2> 199391 INFO  (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[51AD4128F49876FC]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testJmxOnCoreReload
   [junit4]   2> 199410 INFO  (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[51AD4128F49876FC]) [    ] o.a.s.c.TestJmxIntegration Before Reload: Size of infoRegistry: 107 MBeans: 0
   [junit4]   2> 199410 INFO  (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[51AD4128F49876FC]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testJmxOnCoreReload
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestJmxIntegration -Dtests.method=testJmxOnCoreReload -Dtests.seed=51AD4128F49876FC -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=es-ES -Dtests.timezone=Pacific/Funafuti -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.02s J2 | TestJmxIntegration.testJmxOnCoreReload <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([51AD4128F49876FC:53149A4487DBAA31]:0)
   [junit4]    > 	at org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload(TestJmxIntegration.java:180)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 199416 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[51AD4128F49876FC]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testJmxUpdate
   [junit4]   2> 199425 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[51AD4128F49876FC]) [    ] o.a.s.c.TestJmxIntegration Mbeans in server: [solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=handlerStart, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Survivor-Space,name=max, java.lang:type=Threading, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=requests, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=handlerStart, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=errors, solr:dom1=jvm,name0=os,name=systemLoadAverage, solr:dom1=core,dom2=collection1,category=QUERY,scope=/get,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=requestTimes, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=requestTimes, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/health,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=shardRequests, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.updateExecutor.running, solr:dom1=node,category=QUERY,scope=httpShardHandler,name=threadPool.httpShardExecutor.duration, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=handlerStart, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=serverErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=handlerStart, solr:dom1=jvm,name0=buffers,name1=mapped,name=TotalCapacity, solr:dom1=node,category=ADMIN,scope=/admin/health,name=errors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=totalTime, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=rollbacks, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.recoveryExecutor.submitted, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=requestTimes, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/system,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=clientErrors, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreContainerWorkExecutor.running, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=requestTimes, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=reader, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragmenter,name=gap.requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=requestTimes, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrBoundaryScanner,name=breakIterator.requests, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=totalTime, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=errors, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Survivor-Space,name=usage, solr:dom1=jvm,name0=memory,name1=pools,name2=Compressed-Class-Space,name=max, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=clientErrors, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Old-Gen,name=init, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=errors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/file,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=shardRequests, solr:dom1=core,dom2=collection1,category=TLOG,name=state, solr:dom1=core,dom2=collection1,category=CORE,scope=fs,name=spins, solr:dom1=jvm,name0=buffers,name1=direct,name=MemoryUsed, solr:dom1=jvm,name0=classes,name=unloaded, solr:dom1=jvm,name0=threads,name1=waiting,name=count, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=errors, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=threadPool.parallelCoreAdminExecutor.submitted, solr:dom1=core,dom2=collection1,category=TLOG,scope=replay,name=remaining.bytes, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=timeouts, solr:dom1=node,category=CONTAINER,scope=cores,name=loaded, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=handlerStart, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=totalTime, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=new,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=timeouts, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/config,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=handlerStart, solr:dom1=node,category=QUERY,scope=httpShardHandler,name=threadPool.httpShardExecutor.completed, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=clientErrors, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=serverErrors, java.lang:type=Runtime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=cumulativeDeletesById, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=cumulativeDeletesByQuery, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=errors, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.recoveryExecutor.duration, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=timeouts, solr:dom1=core,dom2=collection1,category=CORE,scope=fs,name=totalSpace, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=handlerStart, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreContainerWorkExecutor.duration, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=clientErrors, solr_34095:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=node,category=CONTAINER,scope=cores,name=unloaded, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=serverErrors, java.lang:type=MemoryPool,name=PS Eden Space, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=timeouts, solr:dom1=core,dom2=collection1,category=CORE,scope=fs,name=usableSpace, solr:dom1=jvm,name0=os,name=totalPhysicalMemorySize, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=serverErrors, solr:dom1=jvm,name0=os,name=totalSwapSpaceSize, solr:dom1=jvm,name0=memory,name1=pools,name2=Code-Cache,name=init, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=cumulativeErrors, java.lang:type=MemoryPool,name=PS Survivor Space, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=handlerStart, solr:dom1=jvm,name0=os,name=availableProcessors, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.recoveryExecutor.running, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=errors, solr:dom1=jvm,name0=memory,name1=pools,name2=Compressed-Class-Space,name=used, solr:dom1=node,category=CONTAINER,scope=fs,name=path, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=timeouts, solr:dom1=jvm,name0=threads,name=count, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=timeouts, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=caching, solr:dom1=node,category=ADMIN,scope=/admin/health,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=serverErrors, solr:dom1=jvm,name0=memory,name1=total,name=used, solr:dom1=jvm,name0=memory,name1=pools,name2=Code-Cache,name=usage, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFormatter,name=html.requests, solr:dom1=node,category=CONTAINER,scope=fs,name=spins, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=deletesByQuery, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=serverErrors, solr:dom1=jvm,name0=threads,name1=new,name=count, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=handlerStart, solr:dom1=core,dom2=collection1,category=TLOG,scope=replay,name=ops, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=serverErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=serverErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=timeouts, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Old-Gen,name=committed, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=totalTime, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=openedAt, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=requestTimes, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=initCount, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=errors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=handlerStart, solr:dom1=node,category=QUERY,scope=httpShardHandler,name=leasedConnections, solr:dom1=core,dom2=collection1,category=QUERY,scope=/select,name=requests, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=softAutoCommits, solr:dom1=jvm,name0=memory,name1=non-heap,name=used, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=serverErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=clientErrors, java.lang:type=MemoryPool,name=Code Cache, solr:dom1=core,dom2=collection1,category=CORE,name=refCount, java.nio:type=BufferPool,name=direct, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json,name=totalTime, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=errors, solr:dom1=jvm,name0=threads,name1=daemon,name=count, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=serverErrors, solr:dom1=node,category=CONTAINER,scope=fs,name=coreRoot.path, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=errors, solr:dom1=node,category=ADMIN,scope=/admin/health,name=shardRequests, solr:dom1=jvm,name0=memory,name1=non-heap,name=init, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=requestTimes, solr:dom1=core,dom2=collection1,category=CACHE,scope=searcher,name=queryResultCache, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=totalTime, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=handlerStart, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=searcherName, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=requests, solr:dom1=jvm,name0=memory,name1=heap,name=max, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=shardRequests, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreLoadExecutor.duration, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=shardRequests, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=serverErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=timeouts, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=errors, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=requestTimes, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/file,name=timeouts, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=indexSize, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=errors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=shardRequests, solr:dom1=jvm,name0=buffers,name1=mapped,name=Count, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=errors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=requests, solr:dom1=core,dom2=collection1,categ
   [junit4]   2> ory=ADMIN,scope=/admin/file,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=serverErrors, java.lang:type=Compilation, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.updateExecutor.submitted, solr:dom1=jvm,name0=memory,name1=pools,name2=Compressed-Class-Space,name=committed, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=requests, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=serverErrors, solr:dom1=core,dom2=collection1,category=TLOG,scope=buffered,name=ops, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=adds, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=errors, solr:dom1=jvm,name0=os,name=version, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=serverErrors, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=pendingConnections, solr:dom1=jvm,name0=memory,name1=pools,name2=Metaspace,name=usage, solr:dom1=core,dom2=collection1,category=CACHE,scope=searcher,name=documentCache, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=shardRequests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/plugins,name=totalTime, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=handlerStart, java.lang:type=MemoryPool,name=Compressed Class Space, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=requestTimes, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=requests, solr:dom1=jvm,name0=os,name=committedVirtualMemorySize, solr:dom1=core,dom2=collection1,category=QUERY,scope=/tvrh,name=errors, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Old-Gen,name=usage, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=requests, solr:dom1=jvm,name0=memory,name1=non-heap,name=max, solr:dom1=jvm,name0=os,name=maxFileDescriptorCount, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragmentsBuilder,name=simple.requests, solr:dom1=core,dom2=collection1,category=CORE,name=coreName, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreLoadExecutor.running, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=handlerStart, solr:dom1=jvm,name0=memory,name1=heap,name=used, solr:dom1=core,dom2=collection1,category=CACHE,scope=searcher,name=perSegFilter, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/health,name=totalTime, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=errors, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=confFilesToReplicate, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=totalTime, solr:dom1=node,category=CONTAINER,scope=fs,name=coreRoot.usableSpace, solr:dom1=jvm,name0=os,name=systemCpuLoad, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragListBuilder,name=default.requests, solr:dom1=node,category=ADMIN,scope=/admin/info,name=shardRequests, solr_45985:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=splits, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=handlerStart, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=fetcher, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=requestTimes, solr:dom1=jvm,name0=threads,name1=blocked,name=count, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=requestTimes, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=shardRequests, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=totalTime, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrBoundaryScanner,name=default.requests, solr:dom1=jvm,name0=gc,name1=PS-MarkSweep,name=time, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=totalTime, solr:dom1=core,dom2=collection1,category=TLOG,scope=applyingBuffered,name=ops, solr:dom1=node,category=ADMIN,scope=/admin/info,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=totalTime, solr:dom1=jvm,name0=buffers,name1=direct,name=TotalCapacity, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=serverErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=handlerStart, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=generation, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=errors, java.lang:type=OperatingSystem, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/tvrh,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=clientErrors, solr:dom1=jvm,name0=threads,name1=terminated,name=count, java.lang:type=MemoryManager,name=Metaspace Manager, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=handlerStart, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=shardRequests, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=totalTime, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=errors, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=handlerStart, solr:dom1=node,category=ADMIN,scope=/admin/health,name=requests, solr:dom1=core,dom2=collection1,category=TLOG,scope=replay,name=remaining.logs, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/system,name=requestTimes, solr:dom1=core,dom2=collection1,category=CORE,scope=fs,name=path, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreContainerWorkExecutor.completed, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=totalTime, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=totalTime, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=threadPool.parallelCoreAdminExecutor.duration, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=totalTime, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Eden-Space,name=max, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Old-Gen,name=used, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=serverErrors, solr:dom1=core,dom2=collection1,category=INDEX,name=sizeInBytes, solr:dom1=jvm,name0=memory,name1=pools,name2=Metaspace,name=max, solr:dom1=core,dom2=collection1,category=QUERY,scope=/tvrh,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/info,name=requestTimes, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=commits, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=isSlave, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=serverErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=cumulativeAdds, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/select,name=timeouts, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=timeouts, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=clientErrors, solr:dom1=jvm,name0=os,name=freeSwapSpaceSize, solr:dom1=core,dom2=collection1,category=QUERY,scope=/get,name=totalTime, java.lang:type=MemoryManager,name=CodeCacheManager, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=requests, solr:dom1=node,category=QUERY,scope=httpShardHandler,name=availableConnections, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=indexPath, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=requestTimes, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Eden-Space,name=committed, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=totalTime, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=requestTimes, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Old-Gen,name=used-after-gc, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=timeouts, java.lang:type=GarbageCollector,name=PS Scavenge, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/system,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=totalTime, solr:dom1=node,category=ADMIN,scope=/admin/info,name=clientErrors, solr:dom1=jvm,name0=memory,name1=heap,name=usage, solr:dom1=core,dom2=collection1,category=INDEX,name=size, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/plugins,name=requestTimes, java.nio:type=BufferPool,name=mapped, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Eden-Space,name=used-after-gc, com.sun.management:type=DiagnosticCommand, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=replicationEnabled, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/config,name=serverErrors, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=new,name=maxReached, solr:dom1=node,category=ADMIN,scope=/admin/info,name=totalTime, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/plugins,name=clientErrors, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=threadPool.parallelCoreAdminExecutor.running, solr:dom1=jvm,name0=os,name=arch, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.updateExecutor.duration, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=requests, solr:dom1=jvm,name0=memory,name1=pools,name2=Compressed-Class-Space,name=init, solr:dom1=jvm,name0=os,name=name, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=serverErrors, solr:dom1=node,category=ADMIN,scope=/admin/health,name=handlerStart, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=maxDoc, solr:dom1=jvm,name0=os,name=processCpuLoad, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json,name=errors, solr:dom1=jvm,name0=memory,name1=pools,name2=Metaspace,name=committed, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragmentsBuilder,name=scoreOrder.requests, solr:dom1=node,category=CONTAINER,scope=fs,name=usableSpace, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=serverErrors, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=errors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=merges, solr:dom1=node,category=ADMIN,scope=/admin/health,name=clientErrors, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=totalTime, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=totalTime, solr:dom1=core,dom2=collection1,category=QUERY,scope=/tvrh,name=serverErrors, solr:dom1=node,category=CONTAINER,scope=version,name=specification, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/plugins,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=shardRequests, solr:dom1=jvm,name0=threads,name1=timed_waiting,name=count, solr:dom1=core,dom2=collection1,category=CORE,name=instanceDir, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/tvrh,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/file,name=totalTime, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=serverErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/plugins,name=handlerStart, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=deletedDocs, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=timeouts, solr:dom1=jvm,name0=memory,name1=pools,name2=Metaspace,name=used, solr:dom1=node,category=ADMIN,scope=/admin/info,name=timeouts, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/config,name=requestTimes, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/system,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=serverErrors, JMImplementation:type=MBeanServerDelegate, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=readerDir, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=requestTimes, solr_54617:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=serverErrors, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=requests, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.recoveryExecutor.completed, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=handlerStart, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=deletesById, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=errors, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=new,name=time, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=shardRequests, solr
   [junit4]   2> :dom1=node,category=ADMIN,scope=/admin/collections,name=totalTime, java.lang:type=ClassLoading, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/plugins,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/file,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/select,name=requestTimes, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreContainerWorkExecutor.submitted, java.lang:type=MemoryPool,name=Metaspace, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=clientErrors, solr:dom1=node,category=ADMIN,scope=/admin/health,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=expungeDeletes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/tvrh,name=requestTimes, solr:dom1=core,dom2=collection1,category=CACHE,scope=core,name=fieldCache, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=shardRequests, solr:dom1=core,dom2=collection1,category=CACHE,scope=searcher,name=fieldValueCache, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=totalTime, solr:dom1=core,dom2=collection1,category=QUERY,scope=/select,name=handlerStart, solr:dom1=jvm,name0=memory,name1=pools,name2=Code-Cache,name=max, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/file,name=serverErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=autoCommits, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=requests, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=requestTimes, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=timeouts, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Survivor-Space,name=committed, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=handlerStart, solr:dom1=jvm,name0=gc,name1=PS-MarkSweep,name=count, solr:dom1=node,category=CONTAINER,scope=fs,name=totalSpace, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=errors, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragmenter,name=default.requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/tvrh,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/info,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=timeouts, solr:dom1=jvm,name0=os,name=freePhysicalMemorySize, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=shardRequests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=clientErrors, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=threadPool.parallelCoreAdminExecutor.completed, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=shardRequests, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=errors, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=new,name=warmup, solr:dom1=core,dom2=collection1,category=QUERY,scope=/get,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=requests, solr:dom1=node,category=QUERY,scope=httpShardHandler,name=maxConnections, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=totalTime, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=shardRequests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=totalTime, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragmentsBuilder,name=default.requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=shardRequests, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=docsPending, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=totalTime, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Survivor-Space,name=used-after-gc, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=totalTime, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=totalTime, solr:dom1=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=isMaster, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=handlerStart, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=serverErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json,name=serverErrors, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrEncoder,name=default.requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/config,name=shardRequests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=requestTimes, solr:dom1=jvm,name0=memory,name1=total,name=max, solr:dom1=core,dom2=collection1,category=CORE,name=aliases, solr:dom1=core,dom2=collection1,category=QUERY,scope=/select,name=errors, solr:dom1=jvm,name0=memory,name1=pools,name2=Code-Cache,name=used, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/plugins,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=errors, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=registeredAt, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/get,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/get,name=handlerStart, solr:dom1=jvm,name0=memory,name1=pools,name2=Compressed-Class-Space,name=usage, solr:dom1=node,category=ADMIN,scope=/admin/info,name=serverErrors, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=replicateAfter, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/file,name=requests, solr:dom1=core,dom2=collection1,category=CACHE,scope=searcher,name=filterCache, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=shardRequests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/config,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=handlerStart, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Survivor-Space,name=used, solr:dom1=core,dom2=collection1,category=CORE,name=startTime, java.lang:type=MemoryPool,name=PS Old Gen, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=requestTimes, solr:dom1=jvm,name0=gc,name1=PS-Scavenge,name=time, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreLoadExecutor.completed, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=handlerStart, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Survivor-Space,name=init, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrBoundaryScanner,name=simple.requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=requestTimes, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/file,name=requestTimes, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/select,name=serverErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json,name=shardRequests, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=maxConnections, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=clientErrors, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=availableConnections, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=requestTimes, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=serverErrors, solr:dom1=core,dom2=collection1,category=SEARCHER,name=new, solr:dom1=core,dom2=collection1,category=QUERY,scope=/select,name=shardRequests, solr:dom1=node,category=CONTAINER,scope=version,name=implementation, com.sun.management:type=HotSpotDiagnostic, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=serverErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=requests, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=errors, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreLoadExecutor.submitted, solr:dom1=jvm,name0=memory,name1=non-heap,name=committed, solr:dom1=jvm,name0=buffers,name1=mapped,name=MemoryUsed, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=requests, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=errors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/system,name=clientErrors, java.util.logging:type=Logging, solr:dom1=jvm,name0=memory,name1=pools,name2=Metaspace,name=init, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=totalTime, solr:dom1=jvm,name0=buffers,name1=direct,name=Count, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/select,name=totalTime, solr:dom1=jvm,name0=os,name=processCpuTime, solr:dom1=core,dom2=collection1,category=QUERY,scope=/get,name=serverErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=requestTimes, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/plugins,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=totalTime, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/system,name=totalTime, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/config,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=errors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=requestTimes, solr:dom1=jvm,name0=memory,name1=heap,name=committed, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/file,name=handlerStart, solr:dom1=jvm,name0=threads,name1=deadlock,name=count, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=errors, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=warmupTime, solr:dom1=jvm,name0=threads,name1=runnable,name=count, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/config,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=requestTimes, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=indexVersion, solr:dom1=jvm,name0=memory,name1=heap,name=init, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=numDocs, solr_37454:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=TLOG,scope=copyOverOldUpdates,name=ops, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=requestTimes, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=shardRequests, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFormatter,name=default.requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=errors, solr:dom1=core,dom2=collection1,category=CORE,name=indexDir, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=errors, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=totalTime, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragmenter,name=regex.requests, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/select,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/get,name=requestTimes, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/config,name=requests, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.updateExecutor.completed, java.lang:type=GarbageCollector,name=PS MarkSweep, solr:dom1=core,dom2=collection1,category=QUERY,scope=/get,name=clientErrors, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=totalTime, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=errors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=timeouts, solr:dom1=node,category=QUERY,scope=httpShardHandler,name=threadPool.httpShardExecutor.submitted, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/system,name=errors, solr:dom1=node,category=QUERY,scope=httpShardHandler,name=threadPool.httpShardExecutor.running, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=handlerStart, solr:dom1=node,category=CONTAINER,scope=fs,name=coreRoot.totalSpace, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/system,name=requests, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=totalTime, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Eden-Space,name=usage, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=handlerStart, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=handlerStart, solr:dom1=node,category=CONTAINER,scope=cores,name=lazy, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=errors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=requests, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Eden-Space,name=used, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=optimizes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=shardRequests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=requestTimes, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=index
   [junit4]   2> Version, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Eden-Space,name=init, solr:dom1=core,dom2=collection1,category=QUERY,scope=/tvrh,name=totalTime, solr:dom1=jvm,name0=os,name=openFileDescriptorCount, solr:dom1=node,category=QUERY,scope=httpShardHandler,name=pendingConnections, solr:dom1=jvm,name0=memory,name1=pools,name2=Code-Cache,name=committed, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=errors, solr:dom1=node,category=CACHE,name=fieldCache, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=totalTime, solr:dom1=jvm,name0=memory,name1=total,name=committed, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/plugins,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=shardRequests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=shardRequests, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=timeouts, solr:dom1=node,category=CONTAINER,scope=fs,name=coreRoot.spins, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/system,name=serverErrors, solr:dom1=jvm,name0=threads,name=deadlocks, solr:dom1=core,dom2=collection1,category=QUERY,scope=/get,name=shardRequests, solr_41148:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=errors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/config,name=totalTime, solr:dom1=jvm,name0=memory,name1=non-heap,name=usage, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Old-Gen,name=max, solr:dom1=jvm,name0=classes,name=loaded, solr:dom1=core,dom2=collection1,category=QUERY,scope=/tvrh,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=totalTime, java.lang:type=Memory, solr:dom1=jvm,name0=memory,name1=total,name=init, solr:dom1=jvm,name0=gc,name1=PS-Scavenge,name=count, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=errors, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=leasedConnections, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=requestTimes, solr:dom1=node,category=ADMIN,scope=/admin/info,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=clientErrors, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragListBuilder,name=simple.requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=shardRequests]
   [junit4]   2> 199434 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[51AD4128F49876FC]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[1 (1593873858043052032)]} 0 2
   [junit4]   2> 199436 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[51AD4128F49876FC]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1593873858047246336,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 199436 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[51AD4128F49876FC]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@3a85647f commitCommandVersion:1593873858047246336
   [junit4]   2> 199440 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[51AD4128F49876FC]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@65d2bc71[collection1] main]
   [junit4]   2> 199441 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[51AD4128F49876FC]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 199443 INFO  (searcherExecutor-1006-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@65d2bc71[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C1)))}
   [junit4]   2> 199443 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[51AD4128F49876FC]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 7
   [junit4]   2> 199444 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[51AD4128F49876FC]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testJmxUpdate
   [junit4]   2> 199444 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 199444 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1654011030
   [junit4]   2> 199444 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 199444 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5037cdaa: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@7aafadf8
   [junit4]   2> 199448 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 199448 INFO  (SUITE-TestJmxIntegration-seed#[51AD4128F49876FC]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@20ca8b11: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@44a56387
   [junit4]   2> 199452 INFO  (SUITE-TestJmxIntegration-

[...truncated too long message...]

it4]   2> 980422 INFO  (coreCloseExecutor-5029-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=113307397
   [junit4]   2> 980422 INFO  (coreCloseExecutor-5029-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1ff00cee: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@463db355
   [junit4]   2> 980427 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[51AD4128F49876FC]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5c3b240a{/solr,null,UNAVAILABLE}
   [junit4]   2> 980428 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[51AD4128F49876FC]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 980429 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[51AD4128F49876FC]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@4a0ad46a{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 980429 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[51AD4128F49876FC]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=240514420
   [junit4]   2> 980429 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[51AD4128F49876FC]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 980429 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[51AD4128F49876FC]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@192bb401: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@6dd572a4
   [junit4]   2> 980435 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[51AD4128F49876FC]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 980435 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[51AD4128F49876FC]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@c2d027a: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@3850d2fb
   [junit4]   2> 980440 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[51AD4128F49876FC]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 980440 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[51AD4128F49876FC]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@15556f9c: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@3aa97926
   [junit4]   2> 980440 INFO  (coreCloseExecutor-5031-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@263e7445
   [junit4]   2> 980440 INFO  (coreCloseExecutor-5031-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=641627205
   [junit4]   2> 980440 INFO  (coreCloseExecutor-5031-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6bde7099: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@6a6aecdc
   [junit4]   2> 980448 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[51AD4128F49876FC]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5eceee03{/solr,null,UNAVAILABLE}
   [junit4]   2> 980448 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[51AD4128F49876FC]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_51AD4128F49876FC-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name=Lucene50(blocksize=128), id=Lucene50(blocksize=128), newname=Lucene50(blocksize=128)}, docValues:{}, maxPointsInLeafNode=1926, maxMBSortInHeap=6.003408031293596, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@4ca9a2b5), locale=es-CR, timezone=America/Ojinaga
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=177944448,total=532152320
   [junit4]   2> NOTE: All tests run in this JVM: [ResponseHeaderTest, TestRestoreCore, OverseerTest, TestReversedWildcardFilterFactory, PeerSyncTest, DistribJoinFromCollectionTest, SolrInfoBeanTest, TestCoreAdminApis, TestTrackingShardHandlerFactory, TestRequestForwarding, TestZkChroot, HdfsRestartWhileUpdatingTest, HdfsBasicDistributedZkTest, TestElisionMultitermQuery, TestCharFilters, SystemInfoHandlerTest, DeleteNodeTest, TestSolrCloudWithHadoopAuthPlugin, HdfsLockFactoryTest, TestBinaryResponseWriter, TestLeaderElectionWithEmptyReplica, TestPolicyCloud, DirectoryFactoryTest, TestUtilizeNode, DistributedIntervalFacetingTest, CollectionsAPIAsyncDistributedZkTest, CdcrVersionReplicationTest, TestUseDocValuesAsStored2, SolrGraphiteReporterTest, BasicFunctionalityTest, SimplePostToolTest, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, TestIntervalFaceting, SolrMetricReporterTest, TimeRoutedAliasUpdateProcessorTest, TestConfigSetsAPI, TestScoreJoinQPNoScore, CustomCollectionTest, SignatureUpdateProcessorFactoryTest, TestPushWriter, TestExpandComponent, TestSchemalessBufferedUpdates, TestStressVersions, TestPostingsSolrHighlighter, NodeAddedTriggerTest, TestSolrJ, CloudMLTQParserTest, QueryElevationComponentTest, HdfsTlogReplayBufferedWhileIndexingTest, TestQueryWrapperFilter, PrimUtilsTest, TestSolrIndexConfig, SortByFunctionTest, TestRetrieveFieldsOptimizer, TestEmbeddedSolrServerAdminHandler, TestSystemIdResolver, MoveReplicaTest, SampleTest, QueryParsingTest, TestReload, TestRecovery, HLLUtilTest, DistributedFacetPivotLongTailTest, ResourceLoaderTest, VMParamsZkACLAndCredentialsProvidersTest, TestApiFramework, CoreAdminCreateDiscoverTest, TestSurroundQueryParser, TestSha256AuthenticationProvider, TestXIncludeConfig, TestFreeTextSuggestions, TestDistribStateManager, TestCloudJSONFacetJoinDomain, TermsComponentTest, LeaderInitiatedRecoveryOnShardRestartTest, TestHdfsCloudBackupRestore, LeaderElectionContextKeyTest, ScriptEngineTest, TestTlogReplica, TestRandomFlRTGCloud, TestComplexPhraseQParserPlugin, TestSweetSpotSimilarityFactory, TestSolrCloudWithDelegationTokens, TestNumericTerms32, TestFieldCacheVsDocValues, QueryResultKeyTest, V2ApiIntegrationTest, LoggingHandlerTest, TestFieldCacheReopen, TestJsonFacetRefinement, PrimitiveFieldTypeTest, HighlighterTest, BooleanFieldTest, TestBM25SimilarityFactory, TestConfigSetImmutable, DateRangeFieldTest, TestUnifiedSolrHighlighter, TestStandardQParsers, AddReplicaTest, TestMissingGroups, TestExactSharedStatsCache, TestHdfsBackupRestoreCore, IgnoreCommitOptimizeUpdateProcessorFactoryTest, RemoteQueryErrorTest, ChaosMonkeySafeLeaderTest, AutoScalingHandlerTest, TestFunctionQuery, TestLegacyFieldReuse, TestShortCircuitedRequests, TestTestInjection, InfoHandlerTest, CreateRoutedAliasTest, TestSchemaResource, DocumentBuilderTest, TestObjectReleaseTracker, WrapperMergePolicyFactoryTest, TestAuthorizationFramework, DistributedMLTComponentTest, OutOfBoxZkACLAndCredentialsProvidersTest, ConfigSetsAPITest, DeleteStatusTest, BigEndianAscendingWordDeserializerTest, SuggestComponentContextFilterQueryTest, JsonLoaderTest, TestSerializedLuceneMatchVersion, TestReplicationHandler]
   [junit4] Completed [725/779 (3!)] on J1 in 55.15s, 16 tests, 1 error <<< FAILURES!

[...truncated 7603 lines...]
   [junit4] Suite: org.apache.solr.ltr.TestLTRReRankingPipeline
   [junit4]   2> 8144 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[E7287D79FADCB7DA]) [    ] o.a.s.l.TestLTRReRankingPipeline rerank 1 documents 
   [junit4]   2> 8151 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[E7287D79FADCB7DA]) [    ] o.a.s.l.TestLTRReRankingPipeline doc 0 in pos 0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTRReRankingPipeline -Dtests.method=testDifferentTopN -Dtests.seed=E7287D79FADCB7DA -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=lt -Dtests.timezone=Etc/GMT-3 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.13s J0 | TestLTRReRankingPipeline.testDifferentTopN <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1.0> but was:<0.0>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E7287D79FADCB7DA:16890F29CF677D48]:0)
   [junit4]    > 	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {field=PostingsFormat(name=LuceneFixedGap), id=Lucene50(blocksize=128), final-score=Lucene50(blocksize=128)}, docValues:{final-score=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=96, maxMBSortInHeap=7.198914064201817, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@3e558333), locale=lt, timezone=Etc/GMT-3
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=188776840,total=301465600
   [junit4]   2> NOTE: All tests run in this JVM: [TestRankingFeature, TestLTRReRankingPipeline]
   [junit4] Completed [4/37 (1!)] on J0 in 0.26s, 3 tests, 1 failure <<< FAILURES!

[...truncated 40397 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 25 - Failure

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/25/

No tests ran.

Build Log:
[...truncated 6 lines...]
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://git.apache.org/lucene-solr.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:862)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
	at hudson.scm.SCM.checkout(SCM.java:495)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1724)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git://git.apache.org/lucene-solr.git" returned status code 4:
stdout: 
stderr: error: failed to write new configuration file .git/config.lock

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1996)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1964)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1960)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1597)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1609)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1243)
	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
	at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:922)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:896)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:853)
	at hudson.remoting.UserRequest.perform(UserRequest.java:207)
	at hudson.remoting.UserRequest.perform(UserRequest.java:53)
	at hudson.remoting.Request$2.run(Request.java:358)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
		at hudson.remoting.UserResponse.retrieve(UserRequest.java:310)
		at hudson.remoting.Channel.call(Channel.java:908)
		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:281)
		at com.sun.proxy.$Proxy110.setRemoteUrl(Unknown Source)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:850)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
		at hudson.scm.SCM.checkout(SCM.java:495)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1724)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://git.apache.org/lucene-solr.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:862)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
	at hudson.scm.SCM.checkout(SCM.java:495)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1724)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git://git.apache.org/lucene-solr.git" returned status code 4:
stdout: 
stderr: error: failed to write new configuration file .git/config.lock

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1996)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1964)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1960)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1597)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1609)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1243)
	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
	at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:922)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:896)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:853)
	at hudson.remoting.UserRequest.perform(UserRequest.java:207)
	at hudson.remoting.UserRequest.perform(UserRequest.java:53)
	at hudson.remoting.Request$2.run(Request.java:358)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
		at hudson.remoting.UserResponse.retrieve(UserRequest.java:310)
		at hudson.remoting.Channel.call(Channel.java:908)
		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:281)
		at com.sun.proxy.$Proxy110.setRemoteUrl(Unknown Source)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:850)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
		at hudson.scm.SCM.checkout(SCM.java:495)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1724)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://git.apache.org/lucene-solr.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:862)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
	at hudson.scm.SCM.checkout(SCM.java:495)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1724)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git://git.apache.org/lucene-solr.git" returned status code 4:
stdout: 
stderr: error: failed to write new configuration file .git/config.lock

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1996)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1964)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1960)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1597)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1609)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1243)
	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
	at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:922)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:896)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:853)
	at hudson.remoting.UserRequest.perform(UserRequest.java:207)
	at hudson.remoting.UserRequest.perform(UserRequest.java:53)
	at hudson.remoting.Request$2.run(Request.java:358)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
		at hudson.remoting.UserResponse.retrieve(UserRequest.java:310)
		at hudson.remoting.Channel.call(Channel.java:908)
		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:281)
		at com.sun.proxy.$Proxy110.setRemoteUrl(Unknown Source)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:850)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
		at hudson.scm.SCM.checkout(SCM.java:495)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1724)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Archiving artifacts
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run? 
For example, /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/analysis/common/test/TEST-org.apache.lucene.analysis.ar.TestArabicAnalyzer.xml is 18 hr old

Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 24 - Still unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/24/

3 tests failed.
FAILED:  org.apache.solr.cloud.CollectionPropsTest.testReadWriteCached

Error Message:
Could not see value change after setting collection property. Name: property2, current value: value2, expected value: newValue

Stack Trace:
java.lang.AssertionError: Could not see value change after setting collection property. Name: property2, current value: value2, expected value: newValue
	at __randomizedtesting.SeedInfo.seed([1BCE6473A2A5E68A:FD89A9BD30939A79]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.CollectionPropsTest.waitForValue(CollectionPropsTest.java:146)
	at org.apache.solr.cloud.CollectionPropsTest.testReadWriteCached(CollectionPropsTest.java:115)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.DeleteInactiveReplicaTest.deleteInactiveReplicaTest

Error Message:
Deleted core was still loaded!

Stack Trace:
java.lang.AssertionError: Deleted core was still loaded!
	at __randomizedtesting.SeedInfo.seed([1BCE6473A2A5E68A:D6F0FF80ED049068]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.DeleteInactiveReplicaTest.deleteInactiveReplicaTest(DeleteInactiveReplicaTest.java:86)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
	at __randomizedtesting.SeedInfo.seed([1BCE6473A2A5E68A:C339A097097E24D6]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart(TestReplicationHandler.java:664)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1844 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/test/temp/junit4-J0-20180330_195532_498553079099310282510.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] codec: Asserting, pf: FSTOrd50, dvf: Memory
   [junit4] <<< JVM J0: EOF ----

[...truncated 10244 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionPropsTest
   [junit4]   2> 61084 INFO  (SUITE-CollectionPropsTest-seed#[1BCE6473A2A5E68A]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionPropsTest_1BCE6473A2A5E68A-001/init-core-data-001
   [junit4]   2> 61085 WARN  (SUITE-CollectionPropsTest-seed#[1BCE6473A2A5E68A]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
   [junit4]   2> 61085 INFO  (SUITE-CollectionPropsTest-seed#[1BCE6473A2A5E68A]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 61086 INFO  (SUITE-CollectionPropsTest-seed#[1BCE6473A2A5E68A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 61086 INFO  (SUITE-CollectionPropsTest-seed#[1BCE6473A2A5E68A]-worker) [    ] o.a.s.c.CollectionPropsTest Using legacyCloud?: false
   [junit4]   2> 61087 INFO  (SUITE-CollectionPropsTest-seed#[1BCE6473A2A5E68A]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionPropsTest_1BCE6473A2A5E68A-001/tempDir-001
   [junit4]   2> 61087 INFO  (SUITE-CollectionPropsTest-seed#[1BCE6473A2A5E68A]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 61087 INFO  (Thread-68) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 61087 INFO  (Thread-68) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 61093 ERROR (Thread-68) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 61187 INFO  (SUITE-CollectionPropsTest-seed#[1BCE6473A2A5E68A]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:35243
   [junit4]   2> 61200 INFO  (zkConnectionManagerCallback-102-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61205 INFO  (jetty-launcher-99-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T23:27:37+02:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 61210 INFO  (jetty-launcher-99-thread-3) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T23:27:37+02:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 61210 INFO  (jetty-launcher-99-thread-4) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T23:27:37+02:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 61213 INFO  (jetty-launcher-99-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T23:27:37+02:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 61216 INFO  (jetty-launcher-99-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 61216 INFO  (jetty-launcher-99-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 61216 INFO  (jetty-launcher-99-thread-3) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 61216 INFO  (jetty-launcher-99-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@293efa23{/solr,null,AVAILABLE}
   [junit4]   2> 61217 INFO  (jetty-launcher-99-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 61217 INFO  (jetty-launcher-99-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 61217 INFO  (jetty-launcher-99-thread-1) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 61219 INFO  (jetty-launcher-99-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7d45997b{/solr,null,AVAILABLE}
   [junit4]   2> 61220 INFO  (jetty-launcher-99-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7a7cd67f{HTTP/1.1,[http/1.1]}{127.0.0.1:55609}
   [junit4]   2> 61220 INFO  (jetty-launcher-99-thread-1) [    ] o.e.j.s.Server Started @61256ms
   [junit4]   2> 61220 INFO  (jetty-launcher-99-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55609}
   [junit4]   2> 61220 ERROR (jetty-launcher-99-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 61220 INFO  (jetty-launcher-99-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 61220 INFO  (jetty-launcher-99-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 61220 INFO  (jetty-launcher-99-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 61220 INFO  (jetty-launcher-99-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 61220 INFO  (jetty-launcher-99-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-30T20:23:41.981Z
   [junit4]   2> 61221 INFO  (jetty-launcher-99-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@24d7a586{HTTP/1.1,[http/1.1]}{127.0.0.1:45957}
   [junit4]   2> 61221 INFO  (jetty-launcher-99-thread-3) [    ] o.e.j.s.Server Started @61257ms
   [junit4]   2> 61221 INFO  (jetty-launcher-99-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45957}
   [junit4]   2> 61221 ERROR (jetty-launcher-99-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 61221 INFO  (jetty-launcher-99-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 61221 INFO  (jetty-launcher-99-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 61221 INFO  (jetty-launcher-99-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 61221 INFO  (jetty-launcher-99-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 61221 INFO  (jetty-launcher-99-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-30T20:23:41.982Z
   [junit4]   2> 61233 INFO  (jetty-launcher-99-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 61233 INFO  (jetty-launcher-99-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 61233 INFO  (jetty-launcher-99-thread-4) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 61234 INFO  (jetty-launcher-99-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1d6f71ed{/solr,null,AVAILABLE}
   [junit4]   2> 61237 INFO  (jetty-launcher-99-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 61237 INFO  (jetty-launcher-99-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 61237 INFO  (jetty-launcher-99-thread-2) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 61237 INFO  (jetty-launcher-99-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@32905030{/solr,null,AVAILABLE}
   [junit4]   2> 61238 INFO  (jetty-launcher-99-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@89f172b{HTTP/1.1,[http/1.1]}{127.0.0.1:50641}
   [junit4]   2> 61238 INFO  (jetty-launcher-99-thread-2) [    ] o.e.j.s.Server Started @61274ms
   [junit4]   2> 61238 INFO  (jetty-launcher-99-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50641}
   [junit4]   2> 61238 ERROR (jetty-launcher-99-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 61238 INFO  (jetty-launcher-99-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 61238 INFO  (jetty-launcher-99-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 61238 INFO  (jetty-launcher-99-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 61238 INFO  (jetty-launcher-99-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 61238 INFO  (jetty-launcher-99-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-30T20:23:41.999Z
   [junit4]   2> 61242 INFO  (jetty-launcher-99-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6f577151{HTTP/1.1,[http/1.1]}{127.0.0.1:33332}
   [junit4]   2> 61242 INFO  (jetty-launcher-99-thread-4) [    ] o.e.j.s.Server Started @61279ms
   [junit4]   2> 61242 INFO  (jetty-launcher-99-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33332}
   [junit4]   2> 61243 ERROR (jetty-launcher-99-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 61243 INFO  (jetty-launcher-99-thread-4) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 61243 INFO  (jetty-launcher-99-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 61243 INFO  (jetty-launcher-99-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 61243 INFO  (jetty-launcher-99-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 61243 INFO  (jetty-launcher-99-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-30T20:23:42.004Z
   [junit4]   2> 61244 INFO  (zkConnectionManagerCallback-106-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61244 INFO  (zkConnectionManagerCallback-104-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61249 INFO  (zkConnectionManagerCallback-108-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61250 INFO  (jetty-launcher-99-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 61250 INFO  (jetty-launcher-99-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 61250 INFO  (jetty-launcher-99-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 61269 INFO  (zkConnectionManagerCallback-110-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61274 INFO  (jetty-launcher-99-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 61274 INFO  (jetty-launcher-99-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35243/solr
   [junit4]   2> 61275 INFO  (jetty-launcher-99-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35243/solr
   [junit4]   2> 61283 INFO  (jetty-launcher-99-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35243/solr
   [junit4]   2> 61298 INFO  (zkConnectionManagerCallback-122-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61301 INFO  (zkConnectionManagerCallback-119-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61304 INFO  (zkConnectionManagerCallback-116-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61307 INFO  (jetty-launcher-99-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35243/solr
   [junit4]   2> 61308 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10548a0a2fc0004, likely client has closed socket
   [junit4]   2> 61317 INFO  (zkConnectionManagerCallback-128-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61324 INFO  (zkConnectionManagerCallback-130-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61325 INFO  (zkConnectionManagerCallback-126-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61327 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10548a0a2fc000a, likely client has closed socket
   [junit4]   2> 61337 INFO  (zkConnectionManagerCallback-132-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61351 INFO  (zkConnectionManagerCallback-135-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61366 INFO  (jetty-launcher-99-thread-3) [n:127.0.0.1:45957_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 61366 INFO  (jetty-launcher-99-thread-3) [n:127.0.0.1:45957_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:45957_solr
   [junit4]   2> 61367 INFO  (jetty-launcher-99-thread-3) [n:127.0.0.1:45957_solr    ] o.a.s.c.Overseer Overseer (id=73544823687872523-127.0.0.1:45957_solr-n_0000000000) starting
   [junit4]   2> 61367 INFO  (jetty-launcher-99-thread-4) [n:127.0.0.1:33332_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 61368 INFO  (jetty-launcher-99-thread-4) [n:127.0.0.1:33332_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33332_solr
   [junit4]   2> 61422 INFO  (zkCallback-134-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 61422 INFO  (zkCallback-131-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 61429 INFO  (zkConnectionManagerCallback-142-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61431 INFO  (jetty-launcher-99-thread-3) [n:127.0.0.1:45957_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 61431 INFO  (jetty-launcher-99-thread-2) [n:127.0.0.1:50641_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 61432 INFO  (jetty-launcher-99-thread-3) [n:127.0.0.1:45957_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35243/solr ready
   [junit4]   2> 61433 INFO  (jetty-launcher-99-thread-2) [n:127.0.0.1:50641_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 61435 INFO  (jetty-launcher-99-thread-2) [n:127.0.0.1:50641_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 61435 INFO  (jetty-launcher-99-thread-2) [n:127.0.0.1:50641_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50641_solr
   [junit4]   2> 61441 INFO  (jetty-launcher-99-thread-1) [n:127.0.0.1:55609_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 61443 INFO  (OverseerStateUpdate-73544823687872523-127.0.0.1:45957_solr-n_0000000000) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 61446 INFO  (jetty-launcher-99-thread-3) [n:127.0.0.1:45957_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 61446 INFO  (jetty-launcher-99-thread-3) [n:127.0.0.1:45957_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45957_solr
   [junit4]   2> 61466 INFO  (zkCallback-131-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 61466 INFO  (zkCallback-134-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 61467 INFO  (jetty-launcher-99-thread-1) [n:127.0.0.1:55609_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 61469 INFO  (jetty-launcher-99-thread-1) [n:127.0.0.1:55609_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 61469 INFO  (jetty-launcher-99-thread-1) [n:127.0.0.1:55609_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55609_solr
   [junit4]   2> 61470 INFO  (zkCallback-134-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 61477 INFO  (zkCallback-131-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 61488 INFO  (zkCallback-127-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 61513 INFO  (zkCallback-127-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 61515 INFO  (zkCallback-129-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (4)
   [junit4]   2> 61529 INFO  (zkCallback-141-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (4)
   [junit4]   2> 61669 INFO  (jetty-launcher-99-thread-4) [n:127.0.0.1:33332_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33332.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 61676 INFO  (jetty-launcher-99-thread-4) [n:127.0.0.1:33332_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33332.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 61676 INFO  (jetty-launcher-99-thread-4) [n:127.0.0.1:33332_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33332.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 61677 INFO  (jetty-launcher-99-thread-4) [n:127.0.0.1:33332_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionPropsTest_1BCE6473A2A5E68A-001/tempDir-001/node4/.
   [junit4]   2> 61709 INFO  (zkConnectionManagerCallback-146-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61742 INFO  (jetty-launcher-99-thread-4) [n:127.0.0.1:33332_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 61742 INFO  (jetty-launcher-99-thread-4) [n:127.0.0.1:33332_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35243/solr ready
   [junit4]   2> 61971 INFO  (jetty-launcher-99-thread-1) [n:127.0.0.1:55609_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55609.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 61984 INFO  (jetty-launcher-99-thread-1) [n:127.0.0.1:55609_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55609.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 61984 INFO  (jetty-launcher-99-thread-1) [n:127.0.0.1:55609_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55609.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 61985 INFO  (jetty-launcher-99-thread-1) [n:127.0.0.1:55609_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionPropsTest_1BCE6473A2A5E68A-001/tempDir-001/node1/.
   [junit4]   2> 62064 INFO  (zkConnectionManagerCallback-150-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 62076 INFO  (jetty-launcher-99-thread-1) [n:127.0.0.1:55609_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 62076 INFO  (jetty-launcher-99-thread-1) [n:127.0.0.1:55609_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35243/solr ready
   [junit4]   2> 62742 INFO  (jetty-launcher-99-thread-2) [n:127.0.0.1:50641_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50641.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 62760 INFO  (jetty-launcher-99-thread-2) [n:127.0.0.1:50641_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50641.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 62760 INFO  (jetty-launcher-99-thread-2) [n:127.0.0.1:50641_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50641.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 62761 INFO  (jetty-launcher-99-thread-2) [n:127.0.0.1:50641_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionPropsTest_1BCE6473A2A5E68A-001/tempDir-001/node3/.
   [junit4]   2> 62834 INFO  (zkConnectionManagerCallback-154-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 62844 INFO  (jetty-launcher-99-thread-2) [n:127.0.0.1:50641_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 62844 INFO  (jetty-launcher-99-thread-2) [n:127.0.0.1:50641_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35243/solr ready
   [junit4]   2> 63021 INFO  (jetty-launcher-99-thread-3) [n:127.0.0.1:45957_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45957.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 63044 INFO  (jetty-launcher-99-thread-3) [n:127.0.0.1:45957_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45957.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 63045 INFO  (jetty-launcher-99-thread-3) [n:127.0.0.1:45957_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45957.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 63046 INFO  (jetty-launcher-99-thread-3) [n:127.0.0.1:45957_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionPropsTest_1BCE6473A2A5E68A-001/tempDir-001/node2/.
   [junit4]   2> 63136 INFO  (zkConnectionManagerCallback-156-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 63161 INFO  (zkConnectionManagerCallback-160-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 63163 INFO  (SUITE-CollectionPropsTest-seed#[1BCE6473A2A5E68A]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 63163 INFO  (SUITE-CollectionPropsTest-seed#[1BCE6473A2A5E68A]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35243/solr ready
   [junit4]   2> 63286 INFO  (TEST-CollectionPropsTest.testReadWriteCached-seed#[1BCE6473A2A5E68A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testReadWriteCached
   [junit4]   2> 63304 INFO  (qtp853108670-436) [n:127.0.0.1:45957_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&name=CollectionPropsTest22693325713388335&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 63312 INFO  (OverseerThreadFactory-207-thread-1) [    ] o.a.s.c.a.c.CreateCollectionCmd Create collection CollectionPropsTest22693325713388335
   [junit4]   2> 63443 INFO  (OverseerStateUpdate-73544823687872523-127.0.0.1:45957_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"CollectionPropsTest22693325713388335",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"CollectionPropsTest22693325713388335_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:33332/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 63445 INFO  (OverseerStateUpdate-73544823687872523-127.0.0.1:45957_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"CollectionPropsTest22693325713388335",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"CollectionPropsTest22693325713388335_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:50641/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 63460 INFO  (OverseerStateUpdate-73544823687872523-127.0.0.1:45957_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"CollectionPropsTest22693325713388335",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"CollectionPropsTest22693325713388335_shard2_replica_n5",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:45957/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 63467 INFO  (OverseerStateUpdate-73544823687872523-127.0.0.1:45957_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"CollectionPropsTest22693325713388335",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"CollectionPropsTest22693325713388335_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55609/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 63717 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=CollectionPropsTest22693325713388335_shard1_replica_n2&action=CREATE&numShards=2&collection=CollectionPropsTest22693325713388335&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 63742 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=CollectionPropsTest22693325713388335_shard1_replica_n1&action=CREATE&numShards=2&collection=CollectionPropsTest22693325713388335&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 63742 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 63744 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=CollectionPropsTest22693325713388335_shard2_replica_n6&action=CREATE&numShards=2&collection=CollectionPropsTest22693325713388335&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 63745 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=CollectionPropsTest22693325713388335_shard2_replica_n5&action=CREATE&numShards=2&collection=CollectionPropsTest22693325713388335&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 63901 INFO  (zkCallback-131-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 63902 INFO  (zkCallback-129-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 63903 INFO  (zkCallback-131-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 63903 INFO  (zkCallback-127-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 63903 INFO  (zkCallback-134-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 63921 INFO  (zkCallback-129-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 63933 INFO  (zkCallback-127-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 63934 INFO  (zkCallback-134-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 64810 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 64816 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 64819 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 65340 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.s.IndexSchema [CollectionPropsTest22693325713388335_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 65344 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 65344 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'CollectionPropsTest22693325713388335_shard1_replica_n2' using configuration from collection CollectionPropsTest22693325713388335, trusted=true
   [junit4]   2> 65344 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50641.solr.core.CollectionPropsTest22693325713388335.shard1.replica_n2' (registry 'solr.core.CollectionPropsTest22693325713388335.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 65345 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 65345 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.c.SolrCore [[CollectionPropsTest22693325713388335_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionPropsTest_1BCE6473A2A5E68A-001/tempDir-001/node3/CollectionPropsTest22693325713388335_shard1_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionPropsTest_1BCE6473A2A5E68A-001/tempDir-001/node3/./CollectionPropsTest22693325713388335_shard1_replica_n2/data/]
   [junit4]   2> 65354 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 65357 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.s.IndexSchema [CollectionPropsTest22693325713388335_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 65365 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.s.IndexSchema [CollectionPropsTest22693325713388335_shard2_replica_n5] Schema name=minimal
   [junit4]   2> 65365 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.s.IndexSchema [CollectionPropsTest22693325713388335_shard2_replica_n6] Schema name=minimal
   [junit4]   2> 65368 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 65368 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 65368 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'CollectionPropsTest22693325713388335_shard2_replica_n6' using configuration from collection CollectionPropsTest22693325713388335, trusted=true
   [junit4]   2> 65368 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.CoreContainer Creating SolrCore 'CollectionPropsTest22693325713388335_shard2_replica_n5' using configuration from collection CollectionPropsTest22693325713388335, trusted=true
   [junit4]   2> 65368 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55609.solr.core.CollectionPropsTest22693325713388335.shard2.replica_n6' (registry 'solr.core.CollectionPropsTest22693325713388335.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 65368 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45957.solr.core.CollectionPropsTest22693325713388335.shard2.replica_n5' (registry 'solr.core.CollectionPropsTest22693325713388335.shard2.replica_n5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 65368 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 65368 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 65368 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.c.SolrCore [[CollectionPropsTest22693325713388335_shard2_replica_n6] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionPropsTest_1BCE6473A2A5E68A-001/tempDir-001/node1/CollectionPropsTest22693325713388335_shard2_replica_n6], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionPropsTest_1BCE6473A2A5E68A-001/tempDir-001/node1/./CollectionPropsTest22693325713388335_shard2_replica_n6/data/]
   [junit4]   2> 65368 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.SolrCore [[CollectionPropsTest22693325713388335_shard2_replica_n5] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionPropsTest_1BCE6473A2A5E68A-001/tempDir-001/node2/CollectionPropsTest22693325713388335_shard2_replica_n5], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionPropsTest_1BCE6473A2A5E68A-001/tempDir-001/node2/./CollectionPropsTest22693325713388335_shard2_replica_n5/data/]
   [junit4]   2> 65378 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 65378 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'CollectionPropsTest22693325713388335_shard1_replica_n1' using configuration from collection CollectionPropsTest22693325713388335, trusted=true
   [junit4]   2> 65378 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33332.solr.core.CollectionPropsTest22693325713388335.shard1.replica_n1' (registry 'solr.core.CollectionPropsTest22693325713388335.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c042cac
   [junit4]   2> 65379 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 65379 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.SolrCore [[CollectionPropsTest22693325713388335_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionPropsTest_1BCE6473A2A5E68A-001/tempDir-001/node4/CollectionPropsTest22693325713388335_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionPropsTest_1BCE6473A2A5E68A-001/tempDir-001/node4/./CollectionPropsTest22693325713388335_shard1_replica_n1/data/]
   [junit4]   2> 65492 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 65492 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 65494 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 65494 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 65497 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@406b0b60[CollectionPropsTest22693325713388335_shard1_replica_n1] main]
   [junit4]   2> 65498 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 65499 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 65500 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 65500 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1596395540983054336
   [junit4]   2> 65507 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 65507 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 65508 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 65508 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 65508 INFO  (searcherExecutor-219-thread-1-processing-n:127.0.0.1:33332_solr x:CollectionPropsTest22693325713388335_shard1_replica_n1 c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.SolrCore [CollectionPropsTest22693325713388335_shard1_replica_n1] Registered new searcher Searcher@406b0b60[CollectionPropsTest22693325713388335_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 65514 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/CollectionPropsTest22693325713388335/terms/shard1 to Terms{values={core_node3=0}, version=0}
   [junit4]   2> 65536 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@558c22f4[CollectionPropsTest22693325713388335_shard1_replica_n2] main]
   [junit4]   2> 65537 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 65538 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 65539 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 65539 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 65540 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1596395541024997376
   [junit4]   2> 65543 INFO  (searcherExecutor-216-thread-1-processing-n:127.0.0.1:50641_solr x:CollectionPropsTest22693325713388335_shard1_replica_n2 c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.c.SolrCore [CollectionPropsTest22693325713388335_shard1_replica_n2] Registered new searcher Searcher@558c22f4[CollectionPropsTest22693325713388335_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 65550 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/CollectionPropsTest22693325713388335/terms/shard1 to Terms{values={core_node3=0, core_node4=0}, version=1}
   [junit4]   2> 65577 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 65577 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 65578 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 65578 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 65580 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.s.SolrIndexSearcher Opening [Searcher@2418204e[CollectionPropsTest22693325713388335_shard2_replica_n5] main]
   [junit4]   2> 65581 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 65582 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 65582 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 65586 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 65586 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 65587 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 65587 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 65588 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1596395541075329024
   [junit4]   2> 65589 INFO  (searcherExecutor-218-thread-1-processing-n:127.0.0.1:45957_solr x:CollectionPropsTest22693325713388335_shard2_replica_n5 c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.SolrCore [CollectionPropsTest22693325713388335_shard2_replica_n5] Registered new searcher Searcher@2418204e[CollectionPropsTest22693325713388335_shard2_replica_n5] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 65591 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.ZkShardTerms Successful update of terms at /collections/CollectionPropsTest22693325713388335/terms/shard2 to Terms{values={core_node7=0}, version=0}
   [junit4]   2> 65592 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@290b1333[CollectionPropsTest22693325713388335_shard2_replica_n6] main]
   [junit4]   2> 65593 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 65593 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 65594 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 65594 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1596395541081620480
   [junit4]   2> 65597 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9998ms
   [junit4]   2> 65598 INFO  (searcherExecutor-217-thread-1-processing-n:127.0.0.1:55609_solr x:CollectionPropsTest22693325713388335_shard2_replica_n6 c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.c.SolrCore [CollectionPropsTest22693325713388335_shard2_replica_n6] Registered new searcher Searcher@290b1333[CollectionPropsTest22693325713388335_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 65600 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/CollectionPropsTest22693325713388335/terms/shard2 to Terms{values={core_node7=0, core_node8=0}, version=1}
   [junit4]   2> 65700 INFO  (zkCallback-134-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 65700 INFO  (zkCallback-129-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 65700 INFO  (zkCallback-134-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 65700 INFO  (zkCallback-131-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 65700 INFO  (zkCallback-129-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 65700 INFO  (zkCallback-131-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 65701 INFO  (zkCallback-127-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 65701 INFO  (zkCallback-127-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66039 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 66039 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 66039 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33332/solr/CollectionPropsTest22693325713388335_shard1_replica_n1/
   [junit4]   2> 66040 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=CollectionPropsTest22693325713388335_shard1_replica_n1 url=http://127.0.0.1:33332/solr START replicas=[http://127.0.0.1:50641/solr/CollectionPropsTest22693325713388335_shard1_replica_n2/] nUpdates=100
   [junit4]   2> 66049 INFO  (qtp675863651-468) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.c.S.Request [CollectionPropsTest22693325713388335_shard1_replica_n2]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 66098 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 66098 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 66098 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:45957/solr/CollectionPropsTest22693325713388335_shard2_replica_n5/
   [junit4]   2> 66098 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.u.PeerSync PeerSync: core=CollectionPropsTest22693325713388335_shard2_replica_n5 url=http://127.0.0.1:45957/solr START replicas=[http://127.0.0.1:55609/solr/CollectionPropsTest22693325713388335_shard2_replica_n6/] nUpdates=100
   [junit4]   2> 66122 INFO  (qtp67052619-439) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.c.S.Request [CollectionPropsTest22693325713388335_shard2_replica_n6]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=6
   [junit4]   2> 66342 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=CollectionPropsTest22693325713388335_shard1_replica_n1 url=http://127.0.0.1:33332/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 66342 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 66342 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 66342 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 66348 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33332/solr/CollectionPropsTest22693325713388335_shard1_replica_n1/ shard1
   [junit4]   2> 66403 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.u.PeerSync PeerSync: core=CollectionPropsTest22693325713388335_shard2_replica_n5 url=http://127.0.0.1:45957/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 66403 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 66403 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 66403 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 66406 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:45957/solr/CollectionPropsTest22693325713388335_shard2_replica_n5/ shard2
   [junit4]   2> 66508 INFO  (zkCallback-127-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66508 INFO  (zkCallback-127-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66508 INFO  (zkCallback-134-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66508 INFO  (zkCallback-129-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66508 INFO  (zkCallback-134-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66508 INFO  (zkCallback-131-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66508 INFO  (zkCallback-129-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66508 INFO  (zkCallback-131-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66549 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 66552 INFO  (qtp184111179-442) [n:127.0.0.1:33332_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node3 x:CollectionPropsTest22693325713388335_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=CollectionPropsTest22693325713388335_shard1_replica_n1&action=CREATE&numShards=2&collection=CollectionPropsTest22693325713388335&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2810
   [junit4]   2> 66556 INFO  (qtp675863651-444) [n:127.0.0.1:50641_solr c:CollectionPropsTest22693325713388335 s:shard1 r:core_node4 x:CollectionPropsTest22693325713388335_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=CollectionPropsTest22693325713388335_shard1_replica_n2&action=CREATE&numShards=2&collection=CollectionPropsTest22693325713388335&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2838
   [junit4]   2> 66557 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 66560 INFO  (qtp853108670-447) [n:127.0.0.1:45957_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node7 x:CollectionPropsTest22693325713388335_shard2_replica_n5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=CollectionPropsTest22693325713388335_shard2_replica_n5&action=CREATE&numShards=2&collection=CollectionPropsTest22693325713388335&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2814
   [junit4]   2> 66604 INFO  (qtp67052619-433) [n:127.0.0.1:55609_solr c:CollectionPropsTest22693325713388335 s:shard2 r:core_node8 x:CollectionPropsTest22693325713388335_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=CollectionPropsTest22693325713388335_shard2_replica_n6&action=CREATE&numShards=2&collection=CollectionPropsTest22693325713388335&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2861
   [junit4]   2> 66609 INFO  (qtp853108670-436) [n:127.0.0.1:45957_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 66706 INFO  (zkCallback-134-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66706 INFO  (zkCallback-129-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66706 INFO  (zkCallback-134-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66706 INFO  (zkCallback-129-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66707 INFO  (zkCallback-131-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66707 INFO  (zkCallback-131-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66707 INFO  (zkCallback-127-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 66707 INFO  (zkCallback-127-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/state.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 67347 INFO  (OverseerCollectionConfigSetProcessor-73544823687872523-127.0.0.1:45957_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 67609 INFO  (qtp853108670-436) [n:127.0.0.1:45957_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&collection.configName=conf&name=CollectionPropsTest22693325713388335&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=4305
   [junit4]   2> 67614 INFO  (zkCallback-159-thread-1) [    ] o.a.s.c.c.ZkStateReader A collection property change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/CollectionPropsTest22693325713388335/collectionprops.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating...
   [junit4]   2> 67633 INFO  (zkCallback-159-thread-2) [    ] o.a.s.c.c.ZkStateReader A collection property change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/collectionprops.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating...
   [junit4]   2> 67676 INFO  (zkCallback-159-thread-2) [    ] o.a.s.c.c.ZkStateReader A collection property change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/collectionprops.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating...
   [junit4]   2> 67677 INFO  (zkCallback-159-thread-1) [    ] o.a.s.c.c.ZkStateReader A collection property change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/CollectionPropsTest22693325713388335/collectionprops.json] for collection [CollectionPropsTest22693325713388335] has occurred - updating...
   [junit4]   2> 73621 INFO  (TEST-CollectionPropsTest.testReadWriteCached-seed#[1BCE6473A2A5E68A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testReadWriteCached
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionPropsTest -Dtests.method=testReadWriteCached -Dtests.seed=1BCE6473A2A5E68A -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=de -Dtests.timezone=MIT -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 10.5s J2 | CollectionPropsTest.testRe

[...truncated too long message...]

olr c:delDeadColl s:shard1 r:core_node5 x:delDeadColl_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.delDeadColl.shard1.leader, tag=266399bc
   [junit4]   2> 1807585 INFO  (jetty-closer-4443-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2f5fc813{/solr,null,UNAVAILABLE}
   [junit4]   2> 1807585 INFO  (jetty-closer-4443-thread-3) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1807587 INFO  (coreCloseExecutor-7451-thread-1) [n:127.0.0.1:53049_solr c:delDeadColl s:shard2 r:core_node7 x:delDeadColl_shard2_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.delDeadColl.shard2.leader, tag=2f492681
   [junit4]   2> 1807588 INFO  (coreCloseExecutor-7454-thread-1) [n:127.0.0.1:52476_solr c:delDeadColl s:shard1 r:core_node3 x:delDeadColl_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.delDeadColl.shard1.leader, tag=2582575c
   [junit4]   2> 1807589 INFO  (jetty-closer-4443-thread-4) [    ] o.a.s.c.Overseer Overseer (id=73544937112666120-127.0.0.1:52476_solr-n_0000000000) closing
   [junit4]   2> 1807589 INFO  (OverseerStateUpdate-73544937112666120-127.0.0.1:52476_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:52476_solr
   [junit4]   2> 1807589 WARN  (OverseerAutoScalingTriggerThread-73544937112666120-127.0.0.1:52476_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1807591 INFO  (jetty-closer-4443-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1d09aab4{/solr,null,UNAVAILABLE}
   [junit4]   2> 1807591 INFO  (jetty-closer-4443-thread-2) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1807592 INFO  (jetty-closer-4443-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@335a083{/solr,null,UNAVAILABLE}
   [junit4]   2> 1807592 INFO  (jetty-closer-4443-thread-4) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1807593 ERROR (SUITE-DeleteInactiveReplicaTest-seed#[1BCE6473A2A5E68A]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1807593 INFO  (SUITE-DeleteInactiveReplicaTest-seed#[1BCE6473A2A5E68A]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:49148 49148
   [junit4]   2> 1807676 INFO  (Thread-4923) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:49148 49148
   [junit4]   2> 1807677 WARN  (Thread-4923) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	11	/solr/aliases.json
   [junit4]   2> 	9	/solr/clusterprops.json
   [junit4]   2> 	5	/solr/collections/delDeadColl/terms/shard2
   [junit4]   2> 	5	/solr/security.json
   [junit4]   2> 	5	/solr/collections/delDeadColl/terms/shard1
   [junit4]   2> 	4	/solr/configs/conf
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	36	/solr/collections/delDeadColl/state.json
   [junit4]   2> 	11	/solr/clusterstate.json
   [junit4]   2> 	11	/solr/clusterprops.json
   [junit4]   2> 	2	/solr/overseer_elect/election/73544937112666120-127.0.0.1:52476_solr-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	11	/solr/live_nodes
   [junit4]   2> 	11	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_1BCE6473A2A5E68A-001
   [junit4]   2> Mar 30, 2018 8:52:48 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 6 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1499, maxMBSortInHeap=6.15012976099786, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@5b79e766), locale=uk, timezone=America/North_Dakota/New_Salem
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=217532496,total=524812288
   [junit4]   2> NOTE: All tests run in this JVM: [TestGroupingSearch, TestChildDocTransformer, StatsReloadRaceTest, CreateRoutedAliasTest, TestTolerantUpdateProcessorCloud, TestHdfsBackupRestoreCore, HighlighterMaxOffsetTest, TestRawTransformer, EnumFieldTest, TolerantUpdateProcessorTest, BasicAuthStandaloneTest, TestMergePolicyConfig, HdfsRestartWhileUpdatingTest, TestExactStatsCache, TestDeleteCollectionOnDownNodes, BufferStoreTest, TestIBSimilarityFactory, TestRestManager, DebugComponentTest, XmlUpdateRequestHandlerTest, TestOmitPositions, TestComponentsName, FullSolrCloudDistribCmdsTest, TestDistributedSearch, TestShardHandlerFactory, AddSchemaFieldsUpdateProcessorFactoryTest, TestDocTermOrds, ChaosMonkeyNothingIsSafeWithPullReplicasTest, RulesTest, ReplaceNodeTest, CurrencyFieldTypeTest, DirectUpdateHandlerTest, CleanupOldIndexTest, RequestHandlersTest, CollectionsAPIDistributedZkTest, TestMiniSolrCloudClusterSSL, TestCorePropertiesReload, BadCopyFieldTest, TestPayloadScoreQParserPlugin, SpellingQueryConverterTest, DistributedFacetPivotSmallTest, TestSmileRequest, TestRandomRequestDistribution, CollectionStateFormat2Test, TestSchemaManager, TestManagedStopFilterFactory, QueryResultKeyTest, TestSubQueryTransformerDistrib, TemplateUpdateProcessorTest, TestCloudPivotFacet, NodeLostTriggerTest, ZkCLITest, DeleteStatusTest, TestIndexSearcher, FullHLLTest, BinaryUpdateRequestHandlerTest, TestExceedMaxTermLength, ScheduledMaintenanceTriggerTest, CdcrVersionReplicationTest, TestLargeCluster, TestScoreJoinQPScore, SolrPluginUtilsTest, TestReloadDeadlock, AnalysisErrorHandlingTest, TestExtendedDismaxParser, TestRemoteStreaming, TestHighlightDedupGrouping, DistributedQueryComponentOptimizationTest, ZkSolrClientTest, HdfsTlogReplayBufferedWhileIndexingTest, TestCloudDeleteByQuery, TestLMDirichletSimilarityFactory, BigEndianAscendingWordSerializerTest, ZkNodePropsTest, StatsComponentTest, TestSolrXml, SolrJmxReporterTest, TestExportWriter, TestFastOutputStream, DocValuesNotIndexedTest, TestMaxScoreQueryParser, TestPolicyCloud, DocumentAnalysisRequestHandlerTest, PeerSyncWithIndexFingerprintCachingTest, TestCollationField, SpellCheckCollatorTest, DistributedDebugComponentTest, LeaderFailureAfterFreshStartTest, TestSolrFieldCacheBean, SchemaApiFailureTest, TestBinaryField, TestWordDelimiterFilterFactory, LIRRollingUpdatesTest, MBeansHandlerTest, TestCSVResponseWriter, TestPKIAuthenticationPlugin, RequestLoggingTest, UUIDFieldTest, SuggesterWFSTTest, TlogReplayBufferedWhileIndexingTest, CachingDirectoryFactoryTest, PropertiesRequestHandlerTest, HttpTriggerListenerTest, TestCloudSearcherWarming, HdfsBasicDistributedZkTest, MoveReplicaHDFSTest, TestConfigsApi, CollectionReloadTest, TestInPlaceUpdatesStandalone, TestSolrDeletionPolicy1, TestCollectionAPI, TermVectorComponentDistributedTest, OpenExchangeRatesOrgProviderTest, SolrMetricManagerTest, FieldAnalysisRequestHandlerTest, TestFieldResource, TriggerSetPropertiesIntegrationTest, EchoParamsTest, TestPolicyCloud, TestBlobHandler, TestNumericRangeQuery64, HttpPartitionOnCommitTest, TestDynamicFieldCollectionResource, TestLuceneMatchVersion, AutoAddReplicasIntegrationTest, TestStressLucene, UninvertDocValuesMergePolicyTest, OpenCloseCoreStressTest, ExternalFileFieldSortTest, TestRandomFaceting, SimpleFacetsTest, OverseerTest, TestZkChroot, ShardRoutingCustomTest, TestFaceting, TestRecovery, TestHashPartitioner, TestRangeQuery, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, SimplePostToolTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, SuggesterFSTTest, CoreAdminHandlerTest, SolrRequestParserTest, DocValuesTest, SuggesterTest, SpatialFilterTest, TestCSVLoader, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, FieldMutatingUpdateProcessorTest, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, SortByFunctionTest, DistanceFunctionTest, CacheHeaderTest, IndexBasedSpellCheckerTest, TestSurroundQueryParser, DisMaxRequestHandlerTest, TestQueryUtils, TestWriterPerf, DirectSolrSpellCheckerTest, TestQueryTypes, PrimitiveFieldTypeTest, TestValueSourceCache, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, RequiredFieldsTest, TestSolrQueryParser, FastVectorHighlighterTest, LoggingHandlerTest, RegexBoostProcessorTest, TestJmxIntegration, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, TestAnalyzedSuggestions, JsonLoaderTest, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, HighlighterConfigTest, SOLR749Test, TestQuerySenderListener, UpdateParamsTest, TestSolrIndexConfig, TestQuerySenderNoQuery, SolrIndexConfigTest, BadComponentTest, TestStressRecovery, MultiTermTest, TestDocSet, TestSearchPerf, TestElisionMultitermQuery, TestConfig, OutputWriterTest, TestSolrCoreProperties, TestPostingsSolrHighlighter, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCharFilters, SynonymTokenizerTest, TestXIncludeConfig, TestDFRSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, ResourceLoaderTest, URLClassifyProcessorTest, PluginInfoTest, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, TestTolerantSearch, TokenizerChainTest, TestLuceneIndexBackCompat, ConnectionReuseTest, AliasIntegrationTest, ConcurrentCreateRoutedAliasTest, ConfigSetsAPITest, ConnectionManagerTest, CreateCollectionCleanupTest, DeleteInactiveReplicaTest]
   [junit4] Completed [537/795 (3!)] on J0 in 16.10s, 1 test, 1 failure <<< FAILURES!

[...truncated 50356 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 23 - Failure

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/23/

No tests ran.

Build Log:
[...truncated 6 lines...]
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://git.apache.org/lucene-solr.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:862)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
	at hudson.scm.SCM.checkout(SCM.java:495)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1724)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git://git.apache.org/lucene-solr.git" returned status code 4:
stdout: 
stderr: error: failed to write new configuration file .git/config.lock

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1996)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1964)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1960)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1597)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1609)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1243)
	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
	at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:922)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:896)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:853)
	at hudson.remoting.UserRequest.perform(UserRequest.java:207)
	at hudson.remoting.UserRequest.perform(UserRequest.java:53)
	at hudson.remoting.Request$2.run(Request.java:358)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
		at hudson.remoting.UserResponse.retrieve(UserRequest.java:310)
		at hudson.remoting.Channel.call(Channel.java:908)
		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:281)
		at com.sun.proxy.$Proxy110.setRemoteUrl(Unknown Source)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:850)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
		at hudson.scm.SCM.checkout(SCM.java:495)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1724)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://git.apache.org/lucene-solr.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:862)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
	at hudson.scm.SCM.checkout(SCM.java:495)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1724)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git://git.apache.org/lucene-solr.git" returned status code 4:
stdout: 
stderr: error: failed to write new configuration file .git/config.lock

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1996)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1964)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1960)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1597)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1609)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1243)
	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
	at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:922)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:896)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:853)
	at hudson.remoting.UserRequest.perform(UserRequest.java:207)
	at hudson.remoting.UserRequest.perform(UserRequest.java:53)
	at hudson.remoting.Request$2.run(Request.java:358)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
		at hudson.remoting.UserResponse.retrieve(UserRequest.java:310)
		at hudson.remoting.Channel.call(Channel.java:908)
		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:281)
		at com.sun.proxy.$Proxy110.setRemoteUrl(Unknown Source)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:850)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
		at hudson.scm.SCM.checkout(SCM.java:495)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1724)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://git.apache.org/lucene-solr.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:862)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
	at hudson.scm.SCM.checkout(SCM.java:495)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1724)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git://git.apache.org/lucene-solr.git" returned status code 4:
stdout: 
stderr: error: failed to write new configuration file .git/config.lock

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1996)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1964)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1960)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1597)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1609)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1243)
	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
	at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:922)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:896)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:853)
	at hudson.remoting.UserRequest.perform(UserRequest.java:207)
	at hudson.remoting.UserRequest.perform(UserRequest.java:53)
	at hudson.remoting.Request$2.run(Request.java:358)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
		at hudson.remoting.UserResponse.retrieve(UserRequest.java:310)
		at hudson.remoting.Channel.call(Channel.java:908)
		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:281)
		at com.sun.proxy.$Proxy110.setRemoteUrl(Unknown Source)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:850)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
		at hudson.scm.SCM.checkout(SCM.java:495)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1724)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Archiving artifacts
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run? 
For example, /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/analysis/common/test/TEST-org.apache.lucene.analysis.ar.TestArabicAnalyzer.xml is 9 hr 54 min old

Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 22 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/22/

4 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.ScheduledTriggerTest.testTrigger

Error Message:
expected:<3> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<3> but was:<2>
	at __randomizedtesting.SeedInfo.seed([C4BB1DB32268B02E:A7702B31BBA7C303]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.autoscaling.ScheduledTriggerTest.scheduledTriggerTest(ScheduledTriggerTest.java:112)
	at org.apache.solr.cloud.autoscaling.ScheduledTriggerTest.testTrigger(ScheduledTriggerTest.java:65)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
	at __randomizedtesting.SeedInfo.seed([C4BB1DB32268B02E:1C4CD95789B37272]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart(TestReplicationHandler.java:664)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication

Error Message:
Index: 0, Size: 0

Stack Trace:
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
	at __randomizedtesting.SeedInfo.seed([C4BB1DB32268B02E:D0F346E6016F0D30]:0)
	at java.util.ArrayList.rangeCheck(ArrayList.java:653)
	at java.util.ArrayList.get(ArrayList.java:429)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication(TestReplicationHandler.java:562)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelExecutorStream

Error Message:
Error from server at https://127.0.0.1:34778/solr: Could not fully remove collection: workQueue1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:34778/solr: Could not fully remove collection: workQueue1
	at __randomizedtesting.SeedInfo.seed([35DAD645E5715FBF:88CDA35CDC5D62E2]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1105)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:885)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:818)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelExecutorStream(StreamExpressionTest.java:9726)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1838 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/test/temp/junit4-J1-20180328_120610_6162108527392014355845.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] codec: HighCompressionCompressingStoredFields, pf: Memory, dvf: Direct
   [junit4] <<< JVM J1: EOF ----

[...truncated 11364 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/init-core-data-001
   [junit4]   2> 1091411 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1091412 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1091412 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1091412 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1091412 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@22ccad16{/solr,null,AVAILABLE}
   [junit4]   2> 1091413 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3b53f36c{HTTP/1.1,[http/1.1]}{127.0.0.1:42375}
   [junit4]   2> 1091413 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.Server Started @1090944ms
   [junit4]   2> 1091413 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=42375}
   [junit4]   2> 1091413 ERROR (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1091413 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1091414 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1091414 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1091414 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1091414 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-28T12:52:03.295Z
   [junit4]   2> 1091414 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/solr.xml
   [junit4]   2> 1091417 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1091417 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1091428 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1091460 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1091468 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1091468 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1091470 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/.
   [junit4]   2> 1091470 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1091470 INFO  (coreLoadExecutor-4512-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1091483 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1091490 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1091491 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1091491 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 1091492 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1091492 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1091492 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1091493 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=12, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1091524 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1091524 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1091525 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=82.9306640625, floorSegmentMB=1.9697265625, forceMergeDeletesPctAllowed=20.41170356737787, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5269364850429016
   [junit4]   2> 1091525 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3e5d5aa3[collection1] main]
   [junit4]   2> 1091526 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/collection1/conf
   [junit4]   2> 1091526 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 1091526 INFO  (coreLoadExecutor-4512-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1091527 INFO  (searcherExecutor-4513-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3e5d5aa3[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1091530 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1091531 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1091531 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1091531 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1091531 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2e64d411{/solr,null,AVAILABLE}
   [junit4]   2> 1091531 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@671d5a5e{HTTP/1.1,[http/1.1]}{127.0.0.1:33500}
   [junit4]   2> 1091531 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.Server Started @1091063ms
   [junit4]   2> 1091532 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=33500}
   [junit4]   2> 1091532 ERROR (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1091532 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1091532 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1091532 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1091532 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1091532 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-28T12:52:03.413Z
   [junit4]   2> 1091532 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-002/solr.xml
   [junit4]   2> 1091536 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1091536 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1091537 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1091574 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1091583 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1091584 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1091586 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-002/.
   [junit4]   2> 1091586 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1091599 INFO  (coreLoadExecutor-4522-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1091607 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1091627 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1091628 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1091628 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 1091629 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1091629 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1091629 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-002/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 1091656 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=12, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1091800 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1091800 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1091801 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=82.9306640625, floorSegmentMB=1.9697265625, forceMergeDeletesPctAllowed=20.41170356737787, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5269364850429016
   [junit4]   2> 1091801 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2e896c50[collection1] main]
   [junit4]   2> 1091801 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-002/collection1/conf
   [junit4]   2> 1091803 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 1091803 INFO  (coreLoadExecutor-4522-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1091804 INFO  (searcherExecutor-4523-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2e896c50[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1091813 INFO  (qtp394366275-17062) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1091816 INFO  (qtp394366275-17062) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 4
   [junit4]   2> 1091818 INFO  (qtp1742053463-17078) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1091822 INFO  (qtp1742053463-17078) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 4
   [junit4]   2> 1091823 INFO  (qtp394366275-17061) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1091823 INFO  (qtp394366275-17061) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@135aed91 commitCommandVersion:0
   [junit4]   2> 1091824 INFO  (qtp394366275-17061) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5ca00bb1[collection1] main]
   [junit4]   2> 1091824 INFO  (qtp394366275-17061) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1091825 INFO  (searcherExecutor-4513-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5ca00bb1[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1091825 INFO  (qtp394366275-17061) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 2
   [junit4]   2> 1091825 INFO  (qtp1742053463-17080) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1091825 INFO  (qtp1742053463-17080) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6146e274 commitCommandVersion:0
   [junit4]   2> 1091826 INFO  (qtp1742053463-17080) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2ebc0a17[collection1] main]
   [junit4]   2> 1091826 INFO  (qtp1742053463-17080) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1091827 INFO  (searcherExecutor-4523-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2ebc0a17[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1091827 INFO  (qtp1742053463-17080) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 2
   [junit4]   2> 1091828 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3b53f36c{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1091828 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1620988663
   [junit4]   2> 1091828 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1091828 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@27cdfb3b: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@5d5816aa
   [junit4]   2> 1091832 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1091832 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7e726d65: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@314506b6
   [junit4]   2> 1091835 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1091835 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@259398a6: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@30926b09
   [junit4]   2> 1091842 INFO  (coreCloseExecutor-4528-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@38c55857
   [junit4]   2> 1091842 INFO  (coreCloseExecutor-4528-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=38c55857
   [junit4]   2> 1091842 INFO  (coreCloseExecutor-4528-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5aded02c: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@7cc8f2bc
   [junit4]   2> 1091861 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@22ccad16{/solr,null,UNAVAILABLE}
   [junit4]   2> 1091861 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1091862 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@671d5a5e{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1091862 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1701970497
   [junit4]   2> 1091862 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1091862 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7e55396e: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@4a00f8f1
   [junit4]   2> 1091868 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1091868 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@34a79874: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@314506b6
   [junit4]   2> 1091871 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1091871 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5e6970fd: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@30926b09
   [junit4]   2> 1091872 INFO  (coreCloseExecutor-4530-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3e74e531
   [junit4]   2> 1091872 INFO  (coreCloseExecutor-4530-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=3e74e531
   [junit4]   2> 1091872 INFO  (coreCloseExecutor-4530-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@57b80c11: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@235d7593
   [junit4]   2> 1091883 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2e64d411{/solr,null,UNAVAILABLE}
   [junit4]   2> 1091883 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1091884 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1091885 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1091885 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1091885 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1091885 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@44f2392c{/solr,null,AVAILABLE}
   [junit4]   2> 1091885 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@35c0643c{HTTP/1.1,[http/1.1]}{127.0.0.1:33639}
   [junit4]   2> 1091885 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.Server Started @1091417ms
   [junit4]   2> 1091886 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=33639}
   [junit4]   2> 1091886 ERROR (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1091886 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1091886 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1091886 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1091886 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1091886 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-28T12:52:03.767Z
   [junit4]   2> 1091887 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/solr.xml
   [junit4]   2> 1091890 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1091890 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1091891 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1091923 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1091931 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1091931 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1091933 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/.
   [junit4]   2> 1091933 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1091933 INFO  (coreLoadExecutor-4537-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1091944 INFO  (coreLoadExecutor-4537-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1091951 INFO  (coreLoadExecutor-4537-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1091953 INFO  (coreLoadExecutor-4537-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1091953 INFO  (coreLoadExecutor-4537-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 1091953 INFO  (coreLoadExecutor-4537-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1091953 INFO  (coreLoadExecutor-4537-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1091953 INFO  (coreLoadExecutor-4537-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1091955 INFO  (coreLoadExecutor-4537-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6024585359127814]
   [junit4]   2> 1091996 INFO  (coreLoadExecutor-4537-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1091996 INFO  (coreLoadExecutor-4537-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1091997 INFO  (coreLoadExecutor-4537-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=11, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=91.8203125, floorSegmentMB=0.708984375, forceMergeDeletesPctAllowed=27.572101686061366, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7414537524496233
   [junit4]   2> 1091997 INFO  (coreLoadExecutor-4537-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@37ade85a[collection1] main]
   [junit4]   2> 1091998 INFO  (coreLoadExecutor-4537-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/collection1/conf
   [junit4]   2> 1091998 INFO  (coreLoadExecutor-4537-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1092000 INFO  (searcherExecutor-4538-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@37ade85a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1092008 INFO  (qtp1808009576-17105) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 0
   [junit4]   2> 1092009 INFO  (qtp1808009576-17108) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 0
   [junit4]   2> 1092009 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 1092010 INFO  (qtp1808009576-17108) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[3]} 0 0
   [junit4]   2> 1092010 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[4]} 0 0
   [junit4]   2> 1092011 INFO  (qtp1808009576-17108) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[5]} 0 0
   [junit4]   2> 1092018 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[6]} 0 4
   [junit4]   2> 1092023 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[7]} 0 0
   [junit4]   2> 1092024 INFO  (qtp1808009576-17108) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[8]} 0 0
   [junit4]   2> 1092025 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9]} 0 0
   [junit4]   2> 1092047 INFO  (qtp1808009576-17108) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[10]} 0 0
   [junit4]   2> 1092049 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[11]} 0 1
   [junit4]   2> 1092049 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[12]} 0 0
   [junit4]   2> 1092049 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[13]} 0 0
   [junit4]   2> 1092050 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[14]} 0 0
   [junit4]   2> 1092050 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[15]} 0 0
   [junit4]   2> 1092050 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[16]} 0 0
   [junit4]   2> 1092051 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[17]} 0 0
   [junit4]   2> 1092055 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[18]} 0 3
   [junit4]   2> 1092055 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[19]} 0 0
   [junit4]   2> 1092056 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[20]} 0 0
   [junit4]   2> 1092061 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[21]} 0 4
   [junit4]   2> 1092061 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[22]} 0 0
   [junit4]   2> 1092062 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[23]} 0 0
   [junit4]   2> 1092063 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[24]} 0 0
   [junit4]   2> 1092063 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[25]} 0 0
   [junit4]   2> 1092064 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[26]} 0 0
   [junit4]   2> 1092064 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[27]} 0 0
   [junit4]   2> 1092064 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[28]} 0 0
   [junit4]   2> 1092074 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[29]} 0 9
   [junit4]   2> 1092075 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[30]} 0 0
   [junit4]   2> 1092076 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[31]} 0 0
   [junit4]   2> 1092076 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[32]} 0 0
   [junit4]   2> 1092076 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[33]} 0 0
   [junit4]   2> 1092077 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[34]} 0 0
   [junit4]   2> 1092077 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[35]} 0 0
   [junit4]   2> 1092087 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[36]} 0 0
   [junit4]   2> 1092088 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[37]} 0 0
   [junit4]   2> 1092088 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[38]} 0 0
   [junit4]   2> 1092088 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[39]} 0 0
   [junit4]   2> 1092089 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[40]} 0 0
   [junit4]   2> 1092089 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[41]} 0 0
   [junit4]   2> 1092098 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[42]} 0 0
   [junit4]   2> 1092100 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[43]} 0 0
   [junit4]   2> 1092100 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[44]} 0 0
   [junit4]   2> 1092101 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[45]} 0 0
   [junit4]   2> 1092101 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[46]} 0 0
   [junit4]   2> 1092101 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[47]} 0 0
   [junit4]   2> 1092102 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[48]} 0 0
   [junit4]   2> 1092102 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[49]} 0 0
   [junit4]   2> 1092103 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[50]} 0 0
   [junit4]   2> 1092104 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[51]} 0 0
   [junit4]   2> 1092104 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[52]} 0 0
   [junit4]   2> 1092104 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[53]} 0 0
   [junit4]   2> 1092105 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[54]} 0 0
   [junit4]   2> 1092105 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[55]} 0 0
   [junit4]   2> 1092106 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[56]} 0 0
   [junit4]   2> 1092107 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[57]} 0 0
   [junit4]   2> 1092107 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[58]} 0 0
   [junit4]   2> 1092107 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[59]} 0 0
   [junit4]   2> 1092108 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[60]} 0 0
   [junit4]   2> 1092114 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[61]} 0 0
   [junit4]   2> 1092115 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[62]} 0 0
   [junit4]   2> 1092115 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[63]} 0 0
   [junit4]   2> 1092117 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[64]} 0 0
   [junit4]   2> 1092117 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[65]} 0 0
   [junit4]   2> 1092127 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[66]} 0 0
   [junit4]   2> 1092127 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[67]} 0 0
   [junit4]   2> 1092127 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[68]} 0 0
   [junit4]   2> 1092129 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[69]} 0 0
   [junit4]   2> 1092129 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[70]} 0 0
   [junit4]   2> 1092129 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[71]} 0 0
   [junit4]   2> 1092130 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[72]} 0 0
   [junit4]   2> 1092130 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[73]} 0 0
   [junit4]   2> 1092130 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[74]} 0 0
   [junit4]   2> 1092131 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[75]} 0 0
   [junit4]   2> 1092132 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[76]} 0 0
   [junit4]   2> 1092133 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[77]} 0 0
   [junit4]   2> 1092133 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[78]} 0 0
   [junit4]   2> 1092142 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[79]} 0 0
   [junit4]   2> 1092143 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[80]} 0 0
   [junit4]   2> 1092144 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[81]} 0 0
   [junit4]   2> 1092145 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[82]} 0 0
   [junit4]   2> 1092145 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[83]} 0 0
   [junit4]   2> 1092146 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[84]} 0 0
   [junit4]   2> 1092146 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[85]} 0 0
   [junit4]   2> 1092146 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[86]} 0 0
   [junit4]   2> 1092147 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[87]} 0 0
   [junit4]   2> 1092148 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[88]} 0 0
   [junit4]   2> 1092148 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[89]} 0 0
   [junit4]   2> 1092149 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[90]} 0 0
   [junit4]   2> 1092149 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[91]} 0 0
   [junit4]   2> 1092152 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[92]} 0 0
   [junit4]   2> 1092153 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[93]} 0 0
   [junit4]   2> 1092154 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[94]} 0 0
   [junit4]   2> 1092154 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[95]} 0 0
   [junit4]   2> 1092155 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[96]} 0 0
   [junit4]   2> 1092155 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[97]} 0 0
   [junit4]   2> 1092156 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[98]} 0 0
   [junit4]   2> 1092156 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[99]} 0 0
   [junit4]   2> 1092156 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1092156 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@27c3fbac commitCommandVersion:0
   [junit4]   2> 1092193 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@c5888ff[collection1] main]
   [junit4]   2> 1092193 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1092194 INFO  (searcherExecutor-4538-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@c5888ff[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):c100)))}
   [junit4]   2> 1092194 INFO  (qtp1808009576-17110) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 37
   [junit4]   2> 1092200 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@35c0643c{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1092200 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2119832034
   [junit4]   2> 1092200 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1092200 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6106ff15: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@9e69058
   [junit4]   2> 1092206 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1092206 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@641fb58e: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@314506b6
   [junit4]   2> 1092209 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1092209 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@54b6ee71: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@30926b09
   [junit4]   2> 1092246 INFO  (coreCloseExecutor-4542-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@2ca31800
   [junit4]   2> 1092246 INFO  (coreCloseExecutor-4542-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=2ca31800
   [junit4]   2> 1092246 INFO  (coreCloseExecutor-4542-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@68eef8a6: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@333c0178
   [junit4]   2> 1092260 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@44f2392c{/solr,null,UNAVAILABLE}
   [junit4]   2> 1092260 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1092262 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1092263 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1092263 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1092263 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1092263 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@515d44c2{/solr,null,AVAILABLE}
   [junit4]   2> 1092264 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@46752853{HTTP/1.1,[http/1.1]}{127.0.0.1:58405}
   [junit4]   2> 1092264 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.Server Started @1091795ms
   [junit4]   2> 1092264 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=58405}
   [junit4]   2> 1092264 ERROR (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1092264 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1092264 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1092264 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1092264 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1092264 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-28T12:52:04.145Z
   [junit4]   2> 1092264 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/solr.xml
   [junit4]   2> 1092267 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1092267 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1092269 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1092313 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1092321 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1092321 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1092323 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/.
   [junit4]   2> 1092323 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1092323 INFO  (coreLoadExecutor-4549-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1092332 INFO  (coreLoadExecutor-4549-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1092351 INFO  (coreLoadExecutor-4549-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1092354 INFO  (coreLoadExecutor-4549-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1092355 INFO  (coreLoadExecutor-4549-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 1092355 INFO  (coreLoadExecutor-4549-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@35b96ff6
   [junit4]   2> 1092355 INFO  (coreLoadExecutor-4549-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1092355 INFO  (coreLoadExecutor-4549-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1092439 INFO  (coreLoadExecutor-4549-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1092439 INFO  (coreLoadExecutor-4549-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1092440 INFO  (coreLoadExecutor-4549-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6024585359127814]
   [junit4]   2> 1092442 INFO  (coreLoadExecutor-4549-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5eba8578[collection1] main]
   [junit4]   2> 1092442 INFO  (coreLoadExecutor-4549-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_C4BB1DB32268B02E-001/solr-instance-001/collection1/conf
   [junit4]   2> 1092442 INFO  (coreLoadExecutor-4549-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1092446 INFO  (searcherExecutor-4550-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5eba8578[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):c100)))}
   [junit4]   2> 1092457 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[C4BB1DB32268B02E]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27

[...truncated too long message...]

2> 249994 INFO  (zkCallback-233-thread-2) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:48594_solr
   [junit4]   2> 249995 INFO  (jetty-closer-203-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@d0247a6{/solr,null,UNAVAILABLE}
   [junit4]   2> 249995 INFO  (jetty-closer-203-thread-4) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 249996 INFO  (coreCloseExecutor-759-thread-1) [n:127.0.0.1:48594_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=2fa79d91
   [junit4]   2> 249997 INFO  (coreCloseExecutor-759-thread-3) [n:127.0.0.1:48594_solr c:destination1 s:shard1 r:core_node3 x:destination1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.destination1.shard1.leader, tag=94e6375
   [junit4]   2> 250000 INFO  (jetty-closer-203-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4b69bed4{/solr,null,UNAVAILABLE}
   [junit4]   2> 250000 INFO  (jetty-closer-203-thread-3) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 250001 ERROR (SUITE-StreamExpressionTest-seed#[35DAD645E5715FBF]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 250001 INFO  (SUITE-StreamExpressionTest-seed#[35DAD645E5715FBF]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41122 41122
   [junit4]   2> 250185 INFO  (Thread-166) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41122 41122
   [junit4]   2> 250185 WARN  (Thread-166) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	103	/solr/aliases.json
   [junit4]   2> 	13	/solr/collections/destinationCollection/terms/shard2
   [junit4]   2> 	13	/solr/collections/destinationCollection/terms/shard1
   [junit4]   2> 	10	/solr/collections/parallelDestinationCollection1/terms/shard2
   [junit4]   2> 	9	/solr/collections/parallelDestinationCollection1/terms/shard1
   [junit4]   2> 	8	/solr/configs/ml
   [junit4]   2> 	7	/solr/collections/parallelDestinationCollection/terms/shard1
   [junit4]   2> 	7	/solr/collections/parallelDestinationCollection/terms/shard2
   [junit4]   2> 	4	/solr/collections/workQueue1/terms/shard2
   [junit4]   2> 	4	/solr/configs/conf
   [junit4]   2> 	4	/solr/collections/modelCollection/terms/shard2
   [junit4]   2> 	4	/solr/collections/modelCollection/terms/shard1
   [junit4]   2> 	4	/solr/collections/destination/terms/shard2
   [junit4]   2> 	4	/solr/collections/destination/terms/shard1
   [junit4]   2> 	4	/solr/collections/checkpointCollection/terms/shard1
   [junit4]   2> 	4	/solr/collections/mainCorpus/terms/shard2
   [junit4]   2> 	4	/solr/security.json
   [junit4]   2> 	4	/solr/collections/mainCorpus/terms/shard1
   [junit4]   2> 	4	/solr/collections/uknownCollection/terms/shard2
   [junit4]   2> 	4	/solr/collections/uknownCollection/terms/shard1
   [junit4]   2> 	4	/solr/collections/workQueue/terms/shard2
   [junit4]   2> 	4	/solr/collections/workQueue/terms/shard1
   [junit4]   2> 	3	/solr/collections/workQueue1/terms/shard1
   [junit4]   2> 	3	/solr/collections/destination1/terms/shard1
   [junit4]   2> 	3	/solr/collections/destination1/terms/shard2
   [junit4]   2> 	3	/solr/collections/mainCorpus1/terms/shard1
   [junit4]   2> 	3	/solr/collections/mainCorpus1/terms/shard2
   [junit4]   2> 	3	/solr/collections/checkpointCollection/terms/shard2
   [junit4]   2> 	3	/solr/collections/collection1/terms/shard1
   [junit4]   2> 	3	/solr/collections/collection1/terms/shard2
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	103	/solr/clusterprops.json
   [junit4]   2> 	103	/solr/clusterstate.json
   [junit4]   2> 	40	/solr/collections/destinationCollection/state.json
   [junit4]   2> 	26	/solr/collections/parallelDestinationCollection1/state.json
   [junit4]   2> 	20	/solr/collections/parallelDestinationCollection/state.json
   [junit4]   2> 	10	/solr/collections/mainCorpus1/state.json
   [junit4]   2> 	10	/solr/collections/destination/state.json
   [junit4]   2> 	10	/solr/collections/destination1/state.json
   [junit4]   2> 	10	/solr/collections/collection1/state.json
   [junit4]   2> 	10	/solr/collections/workQueue1/state.json
   [junit4]   2> 	10	/solr/collections/workQueue/state.json
   [junit4]   2> 	10	/solr/collections/uknownCollection/state.json
   [junit4]   2> 	10	/solr/collections/modelCollection/state.json
   [junit4]   2> 	10	/solr/collections/checkpointCollection/state.json
   [junit4]   2> 	10	/solr/collections/mainCorpus/state.json
   [junit4]   2> 	2	/solr/overseer_elect/election/73531834834681862-127.0.0.1:52325_solr-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	103	/solr/live_nodes
   [junit4]   2> 	102	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_35DAD645E5715FBF-001
   [junit4]   2> Mar 28, 2018 1:22:56 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {expr_s=PostingsFormat(name=LuceneVarGapFixedInterval), name_s=PostingsFormat(name=MockRandom), terms_ss=PostingsFormat(name=Memory), join2_s=PostingsFormat(name=Memory), field_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), multiDefault=PostingsFormat(name=Memory), subject=PostingsFormat(name=Memory), a1_s=PostingsFormat(name=Memory), a2_s=PostingsFormat(name=MockRandom), level1_s=PostingsFormat(name=LuceneVarGapFixedInterval), body_t=PostingsFormat(name=MockRandom), s_multi=PostingsFormat(name=MockRandom), col_s=PostingsFormat(name=Memory), level2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_ss=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_s=PostingsFormat(name=MockRandom), tv_text=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), term_s=PostingsFormat(name=Memory), a_t=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), text_s=PostingsFormat(name=LuceneVarGapFixedInterval), id=PostingsFormat(name=LuceneVarGapFixedInterval), text=PostingsFormat(name=MockRandom), featureSet_s=PostingsFormat(name=Memory), checkpoint_ss=PostingsFormat(name=MockRandom), test_t=PostingsFormat(name=MockRandom), ident_s=PostingsFormat(name=Memory), whitetok=PostingsFormat(name=MockRandom), side_s=PostingsFormat(name=MockRandom)}, docValues:{expr_s=DocValuesFormat(name=Lucene70), name_s=DocValuesFormat(name=Direct), iteration_i=DocValuesFormat(name=Lucene70), terms_ss=DocValuesFormat(name=Memory), join2_s=DocValuesFormat(name=Memory), field_s=DocValuesFormat(name=Lucene70), multiDefault=DocValuesFormat(name=Memory), price_f=DocValuesFormat(name=Memory), a1_s=DocValuesFormat(name=Memory), intDefault=DocValuesFormat(name=Lucene70), a_f=DocValuesFormat(name=Memory), join1_i=DocValuesFormat(name=Lucene70), falseNegative_i=DocValuesFormat(name=Lucene70), a2_s=DocValuesFormat(name=Direct), a_i=DocValuesFormat(name=Lucene70), join3_i=DocValuesFormat(name=Direct), test_i=DocValuesFormat(name=Memory), falsePositive_i=DocValuesFormat(name=Lucene70), field_i=DocValuesFormat(name=Memory), level1_s=DocValuesFormat(name=Lucene70), s_multi=DocValuesFormat(name=Direct), col_s=DocValuesFormat(name=Memory), level2_s=DocValuesFormat(name=Lucene70), a_ss=DocValuesFormat(name=Lucene70), a_s=DocValuesFormat(name=Direct), score_f=DocValuesFormat(name=Direct), term_s=DocValuesFormat(name=Memory), text_s=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Lucene70), i_multi=DocValuesFormat(name=Lucene70), featureSet_s=DocValuesFormat(name=Memory), trueNegative_i=DocValuesFormat(name=Direct), index_i=DocValuesFormat(name=Memory), b_ls=DocValuesFormat(name=Memory), test_dt=DocValuesFormat(name=Lucene70), out_i=DocValuesFormat(name=Memory), checkpoint_ss=DocValuesFormat(name=Direct), ident_s=DocValuesFormat(name=Memory), order_i=DocValuesFormat(name=Memory), _version_=DocValuesFormat(name=Lucene70), side_s=DocValuesFormat(name=Direct), truePositive_i=DocValuesFormat(name=Lucene70), miles_i=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=1898, maxMBSortInHeap=6.970375609308578, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@155e1e3d), locale=hr-HR, timezone=Asia/Aqtau
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=157966584,total=488112128
   [junit4]   2> NOTE: All tests run in this JVM: [NoOpResponseParserTest, CloudSolrClientTest, TestSolrJErrorHandling, NormalizeEvaluatorTest, TestJavaBinCodec, TestCoreAdmin, TestHash, ArrayEvaluatorTest, SolrParamTest, TestJsonRecordReader, CommonParamsTest, TestTimeSource, ArcCosineEvaluatorTest, GetByIdTest, TestSuggesterResponse, TestCollectionAdminRequest, GreaterThanEqualToEvaluatorTest, HttpSolrClientBuilderTest, LargeVolumeJettyTest, TangentEvaluatorTest, TestBatchUpdate, PowerEvaluatorTest, StreamExpressionToExplanationTest, RegressionEvaluatorTest, SineEvaluatorTest, NamedListTest, CloudSolrClientBuilderTest, CloudSolrClientMultiConstructorTest, JdbcDriverTest, JDBCStreamTest, StreamExpressionTest]
   [junit4] Completed [147/147 (1!)] on J2 in 149.75s, 156 tests, 1 error <<< FAILURES!

[...truncated 48563 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 21 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/21/

1 tests failed.
FAILED:  org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([13B21A44A257644D:3F806DC92E9312C9]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:914)
	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:260)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

	request was:q=cat:cqcvbg31w5mtrxrzvyrq+2qy3iuuou9s1l04e9gup+l537586fmgnspdexar20+yom3z9jl3gimggudqv2i+y16i5ey89f8c79i2p9cv&wt=xml
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:907)
	... 40 more




Build Log:
[...truncated 1846 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/test/temp/junit4-J0-20180327_020728_0271640160060916007609.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] codec: FastDecompressionCompressingStoredFields, pf: LuceneFixedGap, dvf: Asserting
   [junit4] <<< JVM J0: EOF ----

[...truncated 12783 lines...]
   [junit4] Suite: org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_13B21A44A257644D-001/init-core-data-001
   [junit4]   2> 2888951 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[13B21A44A257644D]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 2888986 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[13B21A44A257644D]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2889012 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[13B21A44A257644D]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 2889140 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[13B21A44A257644D]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2889190 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[13B21A44A257644D]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@328b9d38
   [junit4]   2> 2889203 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[13B21A44A257644D]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@328b9d38
   [junit4]   2> 2889203 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[13B21A44A257644D]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@328b9d38
   [junit4]   2> 2889205 INFO  (coreLoadExecutor-9551-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2889205 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 2889240 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2889277 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2889401 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2889409 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 2889409 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@328b9d38
   [junit4]   2> 2889410 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2889410 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_13B21A44A257644D-001/init-core-data-001/]
   [junit4]   2> 2889411 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=22.8603515625, floorSegmentMB=1.6025390625, forceMergeDeletesPctAllowed=11.13449087582678, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7518014646686648
   [junit4]   2> 2889481 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2889481 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2889482 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2889482 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2889483 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=92.1748046875, floorSegmentMB=0.24609375, forceMergeDeletesPctAllowed=10.751303411813895, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1254389085904946
   [junit4]   2> 2889483 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@48222e5a[collection1] main]
   [junit4]   2> 2889484 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 2889494 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 2889542 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2889542 INFO  (searcherExecutor-9552-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 2889543 INFO  (searcherExecutor-9552-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 2889543 INFO  (searcherExecutor-9552-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 2889543 INFO  (searcherExecutor-9552-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 2889543 INFO  (searcherExecutor-9552-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 2889543 INFO  (searcherExecutor-9552-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 2889544 INFO  (searcherExecutor-9552-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 2889544 INFO  (searcherExecutor-9552-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 2889544 INFO  (searcherExecutor-9552-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 2889545 INFO  (searcherExecutor-9552-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@48222e5a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2889546 INFO  (coreLoadExecutor-9551-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1596059865018531840
   [junit4]   2> 2889560 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[13B21A44A257644D]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 2889560 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[13B21A44A257644D]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@15544518[collection1] realtime]
   [junit4]   2> 2889561 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[13B21A44A257644D]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 0
   [junit4]   2> 2889590 WARN  (Thread-4486) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1596059865039503360 actual=1596059865063620609Retrying with new version .... (1)
   [junit4]   2> 2889590 WARN  (Thread-4487) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1596059865063620608 actual=1596059865063620609Retrying with new version .... (1)
   [junit4]   2> 2889590 WARN  (Thread-4489) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1596059865063620609 actual=1596059865064669184Retrying with new version .... (1)
   [junit4]   2> 2889590 WARN  (Thread-4485) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1596059865039503360 actual=1596059865064669184Retrying with new version .... (1)
   [junit4]   2> 2889590 WARN  (Thread-4488) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1596059865063620609 actual=1596059865064669184Retrying with new version .... (1)
   [junit4]   2> 2889596 WARN  (Thread-4489) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1596059865064669185 actual=1596059865070960640Retrying with new version .... (2)
   [junit4]   2> 2889596 WARN  (Thread-4488) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1596059865070960641 actual=1596059865070960642Retrying with new version .... (2)
   [junit4]   2> 2889602 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[13B21A44A257644D]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1596059865077252096,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2889602 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[13B21A44A257644D]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@5bd27ee commitCommandVersion:1596059865077252096
   [junit4]   2> 2889610 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[13B21A44A257644D]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@32365dc5[collection1] main]
   [junit4]   2> 2889611 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[13B21A44A257644D]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2889613 INFO  (searcherExecutor-9552-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@32365dc5[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C10/9:delGen=1)))}
   [junit4]   2> 2889613 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[13B21A44A257644D]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 12
   [junit4]   2> 2889616 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:10&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 2889617 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:cqcvbg31w5mtrxrzvyrq+2qy3iuuou9s1l04e9gup+l537586fmgnspdexar20+yom3z9jl3gimggudqv2i+y16i5ey89f8c79i2p9cv&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=AtomicUpdateProcessorFactoryTest -Dtests.method=testMultipleThreads -Dtests.seed=13B21A44A257644D -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=sv-SE -Dtests.timezone=America/Halifax -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.06s J0 | AtomicUpdateProcessorFactoryTest.testMultipleThreads <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([13B21A44A257644D:3F806DC92E9312C9]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:914)
   [junit4]    > 	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:260)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]    > 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]    > <response>
   [junit4]    > <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]    > </response>
   [junit4]    > 	request was:q=cat:cqcvbg31w5mtrxrzvyrq+2qy3iuuou9s1l04e9gup+l537586fmgnspdexar20+yom3z9jl3gimggudqv2i+y16i5ey89f8c79i2p9cv&wt=xml
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:907)
   [junit4]    > 	... 40 more
   [junit4]   2> 2889630 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1596059865106612224,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2889630 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@5bd27ee commitCommandVersion:1596059865106612224
   [junit4]   2> 2889635 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@6b3a099a[collection1] main]
   [junit4]   2> 2889638 INFO  (searcherExecutor-9552-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6b3a099a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C10/9:delGen=1) Uninverting(_1(8.0.0):C1)))}
   [junit4]   2> 2889638 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2889638 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 8
   [junit4]   2> 2889639 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 2889639 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:human&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 2889640 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Mr&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 2889640 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:20&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 2889640 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=name_s:Virat&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 2889641 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=multiDefault:Delhi&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 2889643 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1596059865120243712,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2889643 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@5bd27ee commitCommandVersion:1596059865120243712
   [junit4]   2> 2889647 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@2aa96bfb[collection1] main]
   [junit4]   2> 2889647 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2889650 INFO  (searcherExecutor-9552-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2aa96bfb[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C10/9:delGen=1) Uninverting(_2(8.0.0):C1)))}
   [junit4]   2> 2889650 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 7
   [junit4]   2> 2889651 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 2889652 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:human&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 2889653 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:animal&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 2889653 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&indent=true&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 2889654 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Mr&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 2889654 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Dr&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 2889654 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:20&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 2889655 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:40&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 2889655 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=name_s:Virat&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 2889655 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[13B21A44A257644D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=multiDefault:Delhi&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 2889656 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[13B21A44A257644D]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1733877642
   [junit4]   2> 2889656 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[13B21A44A257644D]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2889656 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[13B21A44A257644D]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@406c325b: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@666f4e6b
   [junit4]   2> 2889662 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[13B21A44A257644D]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2889662 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[13B21A44A257644D]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@85481ce: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@51b77417
   [junit4]   2> 2889666 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[13B21A44A257644D]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2889666 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[13B21A44A257644D]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6a58c1bb: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@65fe0ba5
   [junit4]   2> 2889666 INFO  (coreCloseExecutor-9557-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7d8b1b8e
   [junit4]   2> 2889666 INFO  (coreCloseExecutor-9557-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=2106268558
   [junit4]   2> 2889666 INFO  (coreCloseExecutor-9557-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@593a6a36: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@6631dcb2
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_13B21A44A257644D-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name_s=FSTOrd50, title_stringNoNorms=Lucene50(blocksize=128), title_stemmed=PostingsFormat(name=MockRandom), multiDefault=PostingsFormat(name=MockRandom), cat=Lucene50(blocksize=128), id=PostingsFormat(name=Memory), text=FSTOrd50, title=PostingsFormat(name=MockRandom), title_lettertok=PostingsFormat(name=Memory)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Direct), range_facet_i_dv=DocValuesFormat(name=Direct), intDvoDefault=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=1149, maxMBSortInHeap=7.404029356900615, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@4580e59c), locale=sv-SE, timezone=America/Halifax
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=162428368,total=526909440
   [junit4]   2> NOTE: All tests run in this JVM: [DocumentAnalysisRequestHandlerTest, TestLargeCluster, PluginInfoTest, JvmMetricsTest, SolrRequestParserTest, SuggesterWFSTTest, TestLegacyTerms, BinaryUpdateRequestHandlerTest, BasicAuthStandaloneTest, BaseCdcrDistributedZkTest, ParsingFieldUpdateProcessorsTest, PropertiesRequestHandlerTest, RequestHandlersTest, TestMergePolicyConfig, AliasIntegrationTest, CdcrReplicationDistributedZkTest, TestCollationField, DistributedQueryComponentOptimizationTest, TestNonDefinedSimilarityFactory, FullHLLTest, LIRRollingUpdatesTest, SolrMetricManagerTest, TestCrossCoreJoin, DefaultValueUpdateProcessorTest, DistributedFacetPivotSmallTest, RulesTest, TokenizerChainTest, TestCSVLoader, SharedFSAutoReplicaFailoverTest, TestRemoteStreaming, DistributedFacetPivotLargeTest, TestCloudNestedDocsSort, SolrShardReporterTest, TestSizeLimitedDistributedMap, TestSmileRequest, TestTrieFacet, SmileWriterTest, TestQuerySenderNoQuery, TestEmbeddedSolrServerConstructors, LeaderFailureAfterFreshStartTest, TestInPlaceUpdatesStandalone, TestNumericRangeQuery64, TestFieldResource, TestSort, TestSolrDeletionPolicy1, ZkCLITest, ComputePlanActionTest, FullSolrCloudDistribCmdsTest, TlogReplayBufferedWhileIndexingTest, TestRestManager, HdfsBasicDistributedZkTest, CollectionsAPIDistributedZkTest, TestWriterPerf, TestDistributedSearch, TestConfigSetImmutable, HttpPartitionOnCommitTest, HdfsRestartWhileUpdatingTest, UtilsToolTest, SpellingQueryConverterTest, TermVectorComponentDistributedTest, ExternalFileFieldSortTest, QueryResultKeyTest, TestLegacyField, TestIndexingPerformance, BufferStoreTest, TestPayloadScoreQParserPlugin, HdfsTlogReplayBufferedWhileIndexingTest, CoreMergeIndexesAdminHandlerTest, OverseerCollectionConfigSetProcessorTest, SolrPluginUtilsTest, TestPullReplica, ZkFailoverTest, TestSolrCloudWithKerberosAlt, TestLeaderElectionZkExpiry, TestWordDelimiterFilterFactory, OutputWriterTest, TestFastOutputStream, CachingDirectoryFactoryTest, TestScoreJoinQPScore, TestIndexSearcher, SimpleMLTQParserTest, TestSolrXml, TestShardHandlerFactory, AnalyticsQueryTest, UninvertDocValuesMergePolicyTest, CollectionStateFormat2Test, LargeFieldTest, TestNumericTerms64, TestBlendedInfixSuggestions, StatsComponentTest, TestBulkSchemaConcurrent, DistributedDebugComponentTest, TestStressLucene, ZkNodePropsTest, TestRTimerTree, TestInitParams, TestLMDirichletSimilarityFactory, TestMiniSolrCloudClusterSSL, TestIBSimilarityFactory, TestDefaultStatsCache, TestBlobHandler, TestExceedMaxTermLength, TestGroupingSearch, EchoParamsTest, TestSolrFieldCacheBean, IndexBasedSpellCheckerTest, TestLuceneMatchVersion, TestSchemaNameResource, TestCollationFieldDocValues, TestFieldCacheWithThreads, UpdateParamsTest, RecoveryAfterSoftCommitTest, TestSchemaManager, TestReloadDeadlock, DirectoryFactoryTest, TestCSVResponseWriter, AddSchemaFieldsUpdateProcessorFactoryTest, TolerantUpdateProcessorTest, TestJavabinTupleStreamParser, TestCloudDeleteByQuery, DeleteStatusTest, SolrGangliaReporterTest, SortSpecParsingTest, TestFieldCacheSort, SolrIndexSplitterTest, TestRawTransformer, ZkSolrClientTest, TestFaceting, TestHashPartitioner, TestRealTimeGet, TestStressReorder, TestReload, TestStressVersions, SimpleFacetsTest, BadIndexSchemaTest, TestFiltering, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, SpellCheckCollatorTest, CoreAdminHandlerTest, TestFoldingMultitermQuery, SpatialFilterTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, DocValuesMultiTest, DebugComponentTest, CacheHeaderTest, TestSurroundQueryParser, LukeRequestHandlerTest, DisMaxRequestHandlerTest, TestOmitPositions, TermsComponentTest, XmlUpdateRequestHandlerTest, PathHierarchyTokenizerFactoryTest, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, TestSolrQueryParser, LoggingHandlerTest, MBeansHandlerTest, QueryParsingTest, UniqFieldsUpdateProcessorFactoryTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, PingRequestHandlerTest, CSVRequestHandlerTest, TestComponentsName, SearchHandlerTest, HighlighterConfigTest, AlternateDirectoryTest, TestSolrIndexConfig, SolrIndexConfigTest, SampleTest, TestDocSet, TestBinaryField, NumericFieldsTest, MinimalSchemaTest, TestElisionMultitermQuery, TestConfig, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, TestPostingsSolrHighlighter, NotRequiredUniqueKeyTest, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCharFilters, SynonymTokenizerTest, TestSweetSpotSimilarityFactory, TestPerFieldSimilarity, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, ChaosMonkeyNothingIsSafeTest, CleanupOldIndexTest, ConcurrentCreateRoutedAliasTest, CreateCollectionCleanupTest, DeleteLastCustomShardedReplicaTest, DeleteNodeTest, DeleteReplicaTest, DocValuesNotIndexedTest, ForceLeaderTest, MigrateRouteKeyTest, NodeMutatorTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverseerStatusTest, PeerSyncReplicationTest, RestartWhileUpdatingTest, RollingRestartTest, SolrCLIZkUtilsTest, SolrCloudExampleTest, TestCloudJSONFacetJoinDomain, TestCloudRecovery, TestConfigSetsAPIZkFailure, TestDeleteCollectionOnDownNodes, TestDownShardTolerantSearch, TestLeaderElectionWithEmptyReplica, TestRandomRequestDistribution, TestRequestForwarding, TestSSLRandomization, TestSegmentSorting, TestTlogReplica, TestCollectionsAPIViaSolrCloudCluster, TestReplicaProperties, AutoAddReplicasIntegrationTest, ScheduledMaintenanceTriggerTest, TriggerIntegrationTest, TestIntervalFaceting, TestCloudSchemaless, BlockJoinFacetRandomTest, TestNestedDocsSort, TestScoreJoinQPNoScore, BasicAuthIntegrationTest, TestAuthorizationFramework, TestNumericTerms32, VersionInfoTest, AtomicUpdateProcessorFactoryTest]
   [junit4] Completed [768/786 (1!)] on J0 in 0.75s, 4 tests, 1 error <<< FAILURES!

[...truncated 49545 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 20 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/20/

5 tests failed.
FAILED:  org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica

Error Message:
Captured an uncaught exception in thread: Thread[id=6814, name=coreZkRegister-1736-thread-1, state=RUNNABLE, group=TGRP-DeleteReplicaTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6814, name=coreZkRegister-1736-thread-1, state=RUNNABLE, group=TGRP-DeleteReplicaTest]
	at __randomizedtesting.SeedInfo.seed([8D83493A1500EB11:E79528EA7DF2A1DB]:0)
Caused by: java.lang.AssertionError: Failed to delete replica
	at __randomizedtesting.SeedInfo.seed([8D83493A1500EB11]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.DeleteReplicaTest.lambda$raceConditionOnDeleteAndRegisterReplica$4(DeleteReplicaTest.java:228)
	at org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:185)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test

Error Message:
Trigger was not fired 

Stack Trace:
java.lang.AssertionError: Trigger was not fired 
	at __randomizedtesting.SeedInfo.seed([8D83493A1500EB11:5D776E0BBFC86E9]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test(SystemLogListenerTest.java:151)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue

Error Message:
action wasn't interrupted

Stack Trace:
java.lang.AssertionError: action wasn't interrupted
	at __randomizedtesting.SeedInfo.seed([8D83493A1500EB11:44360B941C672DE4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue(TriggerIntegrationTest.java:752)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testSearchRate

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
	at __randomizedtesting.SeedInfo.seed([8D83493A1500EB11:D0CB57B3DAC64D5E]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testSearchRate(TestLargeCluster.java:576)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
	at __randomizedtesting.SeedInfo.seed([8D83493A1500EB11:329687C496EA8E97]:0)
	at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:108)
	at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.doTestNodeLost(TestLargeCluster.java:479)
	at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost(TestLargeCluster.java:320)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1853 lines...]
   [junit4] JVM J0: stdout was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/test/temp/junit4-J0-20180326_001712_445497434041490121435.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] codec: FastDecompressionCompressingStoredFields, pf: Lucene50, dvf: Lucene70
   [junit4] <<< JVM J0: EOF ----

[...truncated 10347 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.SystemLogListenerTest
   [junit4]   2> 414661 INFO  (SUITE-SystemLogListenerTest-seed#[8D83493A1500EB11]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_8D83493A1500EB11-001/init-core-data-001
   [junit4]   2> 414661 WARN  (SUITE-SystemLogListenerTest-seed#[8D83493A1500EB11]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=377 numCloses=377
   [junit4]   2> 414662 INFO  (SUITE-SystemLogListenerTest-seed#[8D83493A1500EB11]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 414680 INFO  (SUITE-SystemLogListenerTest-seed#[8D83493A1500EB11]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 414682 INFO  (SUITE-SystemLogListenerTest-seed#[8D83493A1500EB11]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_8D83493A1500EB11-001/tempDir-001
   [junit4]   2> 414683 INFO  (SUITE-SystemLogListenerTest-seed#[8D83493A1500EB11]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 414683 INFO  (Thread-136) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 414683 INFO  (Thread-136) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 414793 ERROR (Thread-136) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 414834 INFO  (SUITE-SystemLogListenerTest-seed#[8D83493A1500EB11]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:44997
   [junit4]   2> 414976 INFO  (zkConnectionManagerCallback-218-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 415122 INFO  (jetty-launcher-215-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 415143 INFO  (jetty-launcher-215-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 415236 INFO  (jetty-launcher-215-thread-3) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 415306 INFO  (jetty-launcher-215-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 415306 INFO  (jetty-launcher-215-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 415306 INFO  (jetty-launcher-215-thread-3) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 415306 INFO  (jetty-launcher-215-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@f703e6d{/solr,null,AVAILABLE}
   [junit4]   2> 415323 INFO  (jetty-launcher-215-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@579a82a8{HTTP/1.1,[http/1.1]}{127.0.0.1:35394}
   [junit4]   2> 415323 INFO  (jetty-launcher-215-thread-3) [    ] o.e.j.s.Server Started @426981ms
   [junit4]   2> 415323 INFO  (jetty-launcher-215-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35394}
   [junit4]   2> 415323 ERROR (jetty-launcher-215-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 415323 INFO  (jetty-launcher-215-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 415323 INFO  (jetty-launcher-215-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 415323 INFO  (jetty-launcher-215-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 415323 INFO  (jetty-launcher-215-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-26T01:56:25.621Z
   [junit4]   2> 415334 INFO  (jetty-launcher-215-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 415334 INFO  (jetty-launcher-215-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 415334 INFO  (jetty-launcher-215-thread-1) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 415334 INFO  (jetty-launcher-215-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@17684aed{/solr,null,AVAILABLE}
   [junit4]   2> 415334 INFO  (jetty-launcher-215-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@30822dfa{HTTP/1.1,[http/1.1]}{127.0.0.1:33202}
   [junit4]   2> 415334 INFO  (jetty-launcher-215-thread-1) [    ] o.e.j.s.Server Started @426993ms
   [junit4]   2> 415335 INFO  (jetty-launcher-215-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33202}
   [junit4]   2> 415335 ERROR (jetty-launcher-215-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 415335 INFO  (jetty-launcher-215-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 415335 INFO  (jetty-launcher-215-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 415335 INFO  (jetty-launcher-215-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 415335 INFO  (jetty-launcher-215-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-26T01:56:25.633Z
   [junit4]   2> 415336 INFO  (jetty-launcher-215-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 415336 INFO  (jetty-launcher-215-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 415337 INFO  (jetty-launcher-215-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 415370 INFO  (jetty-launcher-215-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6a3d22c7{/solr,null,AVAILABLE}
   [junit4]   2> 415372 INFO  (jetty-launcher-215-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2c2c6d5f{HTTP/1.1,[http/1.1]}{127.0.0.1:39950}
   [junit4]   2> 415372 INFO  (jetty-launcher-215-thread-2) [    ] o.e.j.s.Server Started @427031ms
   [junit4]   2> 415373 INFO  (jetty-launcher-215-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39950}
   [junit4]   2> 415373 ERROR (jetty-launcher-215-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 415373 INFO  (jetty-launcher-215-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 415373 INFO  (jetty-launcher-215-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 415373 INFO  (jetty-launcher-215-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 415373 INFO  (jetty-launcher-215-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-26T01:56:25.671Z
   [junit4]   2> 415655 INFO  (zkConnectionManagerCallback-222-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 415655 INFO  (zkConnectionManagerCallback-224-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 415656 INFO  (zkConnectionManagerCallback-220-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 415656 INFO  (jetty-launcher-215-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 415657 INFO  (jetty-launcher-215-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 415983 INFO  (jetty-launcher-215-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 416057 INFO  (jetty-launcher-215-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44997/solr
   [junit4]   2> 416148 INFO  (jetty-launcher-215-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44997/solr
   [junit4]   2> 416394 INFO  (jetty-launcher-215-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44997/solr
   [junit4]   2> 416422 INFO  (zkConnectionManagerCallback-228-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416487 INFO  (zkConnectionManagerCallback-232-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416678 INFO  (zkConnectionManagerCallback-236-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416731 INFO  (zkConnectionManagerCallback-238-thread-1-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416775 INFO  (zkConnectionManagerCallback-240-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 416983 INFO  (zkConnectionManagerCallback-242-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417685 INFO  (jetty-launcher-215-thread-1) [n:127.0.0.1:33202_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 417702 INFO  (jetty-launcher-215-thread-1) [n:127.0.0.1:33202_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33202_solr
   [junit4]   2> 417705 INFO  (jetty-launcher-215-thread-1) [n:127.0.0.1:33202_solr    ] o.a.s.c.Overseer Overseer (id=72343775722012681-127.0.0.1:33202_solr-n_0000000000) starting
   [junit4]   2> 417945 INFO  (zkConnectionManagerCallback-249-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 418079 INFO  (jetty-launcher-215-thread-1) [n:127.0.0.1:33202_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44997/solr ready
   [junit4]   2> 418153 DEBUG (OverseerAutoScalingTriggerThread-72343775722012681-127.0.0.1:33202_solr-n_0000000000) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 418181 INFO  (jetty-launcher-215-thread-1) [n:127.0.0.1:33202_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33202_solr
   [junit4]   2> 418191 DEBUG (OverseerAutoScalingTriggerThread-72343775722012681-127.0.0.1:33202_solr-n_0000000000) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 418276 DEBUG (OverseerAutoScalingTriggerThread-72343775722012681-127.0.0.1:33202_solr-n_0000000000) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: []
   [junit4]   2> 418276 DEBUG (OverseerAutoScalingTriggerThread-72343775722012681-127.0.0.1:33202_solr-n_0000000000) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 418276 DEBUG (OverseerAutoScalingTriggerThread-72343775722012681-127.0.0.1:33202_solr-n_0000000000) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 418380 INFO  (zkCallback-248-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 418400 INFO  (OverseerStateUpdate-72343775722012681-127.0.0.1:33202_solr-n_0000000000) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 418779 DEBUG (OverseerAutoScalingTriggerThread-72343775722012681-127.0.0.1:33202_solr-n_0000000000) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 418815 DEBUG (OverseerAutoScalingTriggerThread-72343775722012681-127.0.0.1:33202_solr-n_0000000000) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 418852 DEBUG (ScheduledTrigger-821-thread-1) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 419035 INFO  (jetty-launcher-215-thread-2) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 419057 INFO  (jetty-launcher-215-thread-2) [n:127.0.0.1:39950_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 419128 INFO  (jetty-launcher-215-thread-2) [n:127.0.0.1:39950_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39950_solr
   [junit4]   2> 419165 INFO  (zkCallback-248-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 419567 INFO  (zkCallback-241-thread-2-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 419685 INFO  (zkCallback-239-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 419937 DEBUG (ScheduledTrigger-821-thread-2) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 420248 INFO  (jetty-launcher-215-thread-1) [n:127.0.0.1:33202_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33202.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3509bb98
   [junit4]   2> 420393 INFO  (jetty-launcher-215-thread-1) [n:127.0.0.1:33202_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33202.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3509bb98
   [junit4]   2> 420416 INFO  (jetty-launcher-215-thread-1) [n:127.0.0.1:33202_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33202.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3509bb98
   [junit4]   2> 420417 INFO  (jetty-launcher-215-thread-1) [n:127.0.0.1:33202_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_8D83493A1500EB11-001/tempDir-001/node1/.
   [junit4]   2> 420558 INFO  (jetty-launcher-215-thread-3) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 420560 INFO  (jetty-launcher-215-thread-3) [n:127.0.0.1:35394_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 420575 INFO  (jetty-launcher-215-thread-3) [n:127.0.0.1:35394_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35394_solr
   [junit4]   2> 420591 INFO  (zkCallback-241-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 420594 INFO  (zkCallback-248-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 420651 INFO  (zkCallback-239-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 420708 INFO  (zkCallback-237-thread-1-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 421160 DEBUG (ScheduledTrigger-821-thread-1) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 421792 INFO  (jetty-launcher-215-thread-2) [n:127.0.0.1:39950_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39950.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3509bb98
   [junit4]   2> 421864 INFO  (jetty-launcher-215-thread-2) [n:127.0.0.1:39950_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39950.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3509bb98
   [junit4]   2> 421864 INFO  (jetty-launcher-215-thread-2) [n:127.0.0.1:39950_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39950.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3509bb98
   [junit4]   2> 421866 INFO  (jetty-launcher-215-thread-2) [n:127.0.0.1:39950_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_8D83493A1500EB11-001/tempDir-001/node2/.
   [junit4]   2> 421951 INFO  (zkConnectionManagerCallback-253-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 422087 INFO  (jetty-launcher-215-thread-2) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 422087 INFO  (jetty-launcher-215-thread-2) [n:127.0.0.1:39950_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44997/solr ready
   [junit4]   2> 422198 DEBUG (ScheduledTrigger-821-thread-3) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 422402 INFO  (jetty-launcher-215-thread-3) [n:127.0.0.1:35394_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35394.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3509bb98
   [junit4]   2> 422434 INFO  (jetty-launcher-215-thread-3) [n:127.0.0.1:35394_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35394.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3509bb98
   [junit4]   2> 422434 INFO  (jetty-launcher-215-thread-3) [n:127.0.0.1:35394_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35394.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3509bb98
   [junit4]   2> 422452 INFO  (jetty-launcher-215-thread-3) [n:127.0.0.1:35394_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_8D83493A1500EB11-001/tempDir-001/node3/.
   [junit4]   2> 422526 INFO  (zkConnectionManagerCallback-257-thread-1-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 422528 INFO  (jetty-launcher-215-thread-3) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 422542 INFO  (jetty-launcher-215-thread-3) [n:127.0.0.1:35394_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44997/solr ready
   [junit4]   2> 422702 INFO  (zkConnectionManagerCallback-259-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 422784 INFO  (zkConnectionManagerCallback-263-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 422797 INFO  (SUITE-SystemLogListenerTest-seed#[8D83493A1500EB11]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 422890 INFO  (SUITE-SystemLogListenerTest-seed#[8D83493A1500EB11]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44997/solr ready
   [junit4]   2> 423172 INFO  (qtp1149748957-1182) [n:127.0.0.1:39950_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=3&name=.system&nrtReplicas=3&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 423224 DEBUG (ScheduledTrigger-821-thread-3) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 423392 INFO  (OverseerThreadFactory-823-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection .system
   [junit4]   2> 423613 INFO  (OverseerStateUpdate-72343775722012681-127.0.0.1:33202_solr-n_0000000000) [n:127.0.0.1:33202_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":".system",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":".system_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:39950/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 423648 INFO  (OverseerStateUpdate-72343775722012681-127.0.0.1:33202_solr-n_0000000000) [n:127.0.0.1:33202_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":".system",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":".system_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:33202/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 423649 INFO  (OverseerStateUpdate-72343775722012681-127.0.0.1:33202_solr-n_0000000000) [n:127.0.0.1:33202_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":".system",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":".system_shard1_replica_n3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:35394/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 424008 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n2&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 424009 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 424010 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n1&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 424010 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 424114 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n3&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 424114 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 424226 DEBUG (ScheduledTrigger-821-thread-4) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 424433 INFO  (zkCallback-239-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 424434 INFO  (zkCallback-239-thread-2-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 424442 INFO  (zkCallback-237-thread-1-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 424442 INFO  (zkCallback-237-thread-2-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 424446 INFO  (zkCallback-241-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 424446 INFO  (zkCallback-241-thread-2-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 425275 DEBUG (ScheduledTrigger-821-thread-1) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 425322 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 425545 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 425580 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
   [junit4]   2> 425598 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.s.IndexSchema [.system_shard1_replica_n1] Schema name=_system collection or core
   [junit4]   2> 425616 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
   [junit4]   2> 425619 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.IndexSchema [.system_shard1_replica_n2] Schema name=_system collection or core
   [junit4]   2> 425649 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
   [junit4]   2> 425654 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
   [junit4]   2> 425654 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 425655 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.s.ManagedIndexSchema Managed schema znode at /configs/.system/managed-schema already exists - no need to create it
   [junit4]   2> 425669 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.ManagedIndexSchema Created and persisted managed schema znode at /configs/.system/managed-schema
   [junit4]   2> 425670 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/.system/schema.xml to /configs/.system/schema.xml.bak
   [junit4]   2> 425670 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore '.system_shard1_replica_n1' using configuration from collection .system, trusted=true
   [junit4]   2> 425670 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39950.solr.core..system.shard1.replica_n1' (registry 'solr.core..system.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3509bb98
   [junit4]   2> 425670 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 425670 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.c.SolrCore [[.system_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_8D83493A1500EB11-001/tempDir-001/node2/.system_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_8D83493A1500EB11-001/tempDir-001/node2/./.system_shard1_replica_n1/data/]
   [junit4]   2> 425697 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, the non-managed schema /configs/.system/schema.xml no longer exists.
   [junit4]   2> 425697 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore '.system_shard1_replica_n2' using configuration from collection .system, trusted=true
   [junit4]   2> 425697 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33202.solr.core..system.shard1.replica_n2' (registry 'solr.core..system.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3509bb98
   [junit4]   2> 425697 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 425697 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.SolrCore [[.system_shard1_replica_n2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_8D83493A1500EB11-001/tempDir-001/node1/.system_shard1_replica_n2], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_8D83493A1500EB11-001/tempDir-001/node1/./.system_shard1_replica_n2/data/]
   [junit4]   2> 425760 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.s.IndexSchema [.system_shard1_replica_n3] Schema name=_system collection or core
   [junit4]   2> 425796 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
   [junit4]   2> 425796 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.c.CoreContainer Creating SolrCore '.system_shard1_replica_n3' using configuration from collection .system, trusted=true
   [junit4]   2> 425797 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35394.solr.core..system.shard1.replica_n3' (registry 'solr.core..system.shard1.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3509bb98
   [junit4]   2> 425797 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 425797 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.c.SolrCore [[.system_shard1_replica_n3] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_8D83493A1500EB11-001/tempDir-001/node3/.system_shard1_replica_n3], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_8D83493A1500EB11-001/tempDir-001/node3/./.system_shard1_replica_n3/data/]
   [junit4]   2> 426001 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 426002 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 426019 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if 1 uncommited docs 
   [junit4]   2> 426019 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 426021 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3e71ab7f[.system_shard1_replica_n1] main]
   [junit4]   2> 426022 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/.system
   [junit4]   2> 426022 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/.system
   [junit4]   2> 426022 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/.system/managed-schema
   [junit4]   2> 426056 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 426056 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 426057 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: if 1 uncommited docs 
   [junit4]   2> 426057 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 426072 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@7f91be4f[.system_shard1_replica_n2] main]
   [junit4]   2> 426073 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/.system
   [junit4]   2> 426073 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/.system
   [junit4]   2> 426073 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/.system/managed-schema
   [junit4]   2> 426129 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 426129 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 426131 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.u.CommitTracker Hard AutoCommit: if 1 uncommited docs 
   [junit4]   2> 426131 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 426143 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 426144 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 426144 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595963496403566592
   [junit4]   2> 426145 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Retrieved schema version 0 from ZooKeeper
   [junit4]   2> 426148 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.s.IndexSchema [.system_shard1_replica_n1] Schema name=_system collection or core
   [junit4]   2> 426161 INFO  (searcherExecutor-831-thread-1-processing-n:127.0.0.1:33202_solr x:.system_shard1_replica_n2 s:shard1 c:.system r:core_node5) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.SolrCore [.system_shard1_replica_n2] Registered new searcher Searcher@7f91be4f[.system_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 426165 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.s.SolrIndexSearcher Opening [Searcher@2c4c6b40[.system_shard1_replica_n3] main]
   [junit4]   2> 432236 WARN  (jetty-launcher-215-thread-1-SendThread(127.0.0.1:44997)) [n:127.0.0.1:33202_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 8586ms for sessionid 0x1010447dd9f000a
   [junit4]   2> 432237 WARN  (SUITE-SystemLogListenerTest-seed#[8D83493A1500EB11]-worker-SendThread(127.0.0.1:44997)) [    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 8668ms for sessionid 0x1010447dd9f000e
   [junit4]   2> 432283 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
   [junit4]   2> 432286 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Finished refreshing schema in 6140 ms
   [junit4]   2> 432286 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 432287 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595963502844968960
   [junit4]   2> 432288 INFO  (searcherExecutor-830-thread-1-processing-n:127.0.0.1:39950_solr x:.system_shard1_replica_n1 s:shard1 c:.system r:core_node4) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.c.SolrCore [.system_shard1_replica_n1] Registered new searcher Searcher@3e71ab7f[.system_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 432247 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1010447dd9f000a, likely client has closed socket
   [junit4]   2> 432308 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1010447dd9f000e, likely client has closed socket
   [junit4]   2> 432247 DEBUG (ScheduledTrigger-821-thread-1) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 432246 WARN  (jetty-launcher-215-thread-2-SendThread(127.0.0.1:44997)) [n:127.0.0.1:39950_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 8729ms for sessionid 0x1010447dd9f000b
   [junit4]   2> 432309 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1010447dd9f000b, likely client has closed socket
   [junit4]   2> 432309 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/.system
   [junit4]   2> 432310 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/.system
   [junit4]   2> 432310 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/.system/managed-schema
   [junit4]   2> 432311 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/.system/terms/shard1 to Terms{values={core_node5=0}, version=0}
   [junit4]   2> 432246 WARN  (jetty-launcher-215-thread-3-SendThread(127.0.0.1:44997)) [n:127.0.0.1:35394_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 8694ms for sessionid 0x1010447dd9f000c
   [junit4]   2> 432320 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1010447dd9f000c, likely client has closed socket
   [junit4]   2> 432359 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=3 found=1 timeoutin=9999ms
   [junit4]   2> 432360 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 432361 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 432372 WARN  (zkConnectionManagerCallback-249-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c72c022 name: ZooKeeperConnection Watcher:127.0.0.1:44997/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 432373 WARN  (zkConnectionManagerCallback-249-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 432373 INFO  (zkCallback-241-thread-2-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 432373 INFO  (zkCallback-241-thread-3-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 432373 INFO  (zkCallback-237-thread-2-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 432373 INFO  (zkCallback-237-thread-1-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 432380 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595963502942486528
   [junit4]   2> 432383 INFO  (searcherExecutor-832-thread-1-processing-n:127.0.0.1:35394_solr x:.system_shard1_replica_n3 s:shard1 c:.system r:core_node6) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.c.SolrCore [.system_shard1_replica_n3] Registered new searcher Searcher@2c4c6b40[.system_shard1_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 432383 INFO  (zkCallback-239-thread-2-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 432383 INFO  (zkCallback-239-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 432394 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.c.ZkShardTerms Successful update of terms at /collections/.system/terms/shard1 to Terms{values={core_node6=0, core_node5=0}, version=1}
   [junit4]   2> 432397 WARN  (zkConnectionManagerCallback-263-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@729a8c0a name: ZooKeeperConnection Watcher:127.0.0.1:44997/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 432397 WARN  (zkConnectionManagerCallback-263-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 432445 WARN  (zkConnectionManagerCallback-257-thread-1-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1a3c40c8 name: ZooKeeperConnection Watcher:127.0.0.1:44997/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 432445 WARN  (zkConnectionManagerCallback-257-thread-1-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 432453 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/.system/terms/shard1 to Terms{values={core_node6=0, core_node4=0, core_node5=0}, version=2}
   [junit4]   2> 432518 WARN  (zkConnectionManagerCallback-253-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@208dfd3a name: ZooKeeperConnection Watcher:127.0.0.1:44997/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 432519 WARN  (zkConnectionManagerCallback-253-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 432860 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 432860 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 432860 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33202/solr/.system_shard1_replica_n2/
   [junit4]   2> 432860 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=.system_shard1_replica_n2 url=http://127.0.0.1:33202/solr START replicas=[http://127.0.0.1:39950/solr/.system_shard1_replica_n1/, http://127.0.0.1:35394/solr/.system_shard1_replica_n3/] nUpdates=100
   [junit4]   2> 432902 INFO  (qtp1149748957-1188) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.c.S.Request [.system_shard1_replica_n1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 432927 INFO  (qtp1351726253-1163) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.c.S.Request [.system_shard1_replica_n3]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 433168 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=.system_shard1_replica_n2 url=http://127.0.0.1:33202/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 433168 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 433168 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 433168 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 433171 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33202/solr/.system_shard1_replica_n2/ shard1
   [junit4]   2> 433298 INFO  (zkCallback-237-thread-1-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433299 INFO  (zkCallback-237-thread-2-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433299 INFO  (zkCallback-239-thread-2-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433299 INFO  (zkCallback-239-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433299 INFO  (zkCallback-241-thread-2-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433299 INFO  (zkCallback-241-thread-3-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433310 DEBUG (ScheduledTrigger-821-thread-4) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 433339 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 433340 INFO  (qtp998215886-1172) [n:127.0.0.1:33202_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n2&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=9332
   [junit4]   2> 433401 INFO  (qtp1351726253-1161) [n:127.0.0.1:35394_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n3&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=9287
   [junit4]   2> 433504 INFO  (zkCallback-237-thread-2-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433507 INFO  (zkCallback-237-thread-1-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433507 INFO  (zkCallback-239-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433507 INFO  (zkCallback-239-thread-2-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433538 INFO  (zkCallback-241-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433538 INFO  (zkCallback-241-thread-2-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433538 INFO  (zkConnectionManagerCallback-257-thread-1-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 433541 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n1&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=9530
   [junit4]   2> 433596 INFO  (qtp1149748957-1182) [n:127.0.0.1:39950_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 433647 INFO  (zkCallback-237-thread-2-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433648 INFO  (zkCallback-237-thread-1-processing-n:127.0.0.1:35394_solr) [n:127.0.0.1:35394_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433648 INFO  (zkCallback-239-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433648 INFO  (zkCallback-239-thread-2-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433649 INFO  (zkCallback-241-thread-2-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433649 INFO  (zkCallback-241-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 433976 WARN  (jetty-launcher-215-thread-1-SendThread(127.0.0.1:44997)) [n:127.0.0.1:33202_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x1010447dd9f000a has expired
   [junit4]   2> 434065 WARN  (zkConnectionManagerCallback-249-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c72c022 name: ZooKeeperConnection Watcher:127.0.0.1:44997/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 434065 WARN  (zkConnectionManagerCallback-249-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 434065 WARN  (zkConnectionManagerCallback-249-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 434105 INFO  (zkConnectionManagerCallback-249-thread-1-processing-n:127.0.0.1:33202_solr-EventThread) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 434105 INFO  (zkConnectionManagerCallback-249-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 434142 INFO  (zkConnectionManagerCallback-249-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 434142 INFO  (zkConnectionManagerCallback-249-thread-1-processing-n:127.0.0.1:33202_solr) [n:127.0.0.1:33202_solr    ] o.a.s.c.c.ConnectionManager zkClient Connected:true
   [junit4]   2> 434171 WARN  (jetty-launcher-215-thread-2-SendThread(127.0.0.1:44997)) [n:127.0.0.1:39950_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x1010447dd9f000b has expired
   [junit4]   2> 434191 WARN  (zkConnectionManagerCallback-253-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@208dfd3a name: ZooKeeperConnection Watcher:127.0.0.1:44997/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 434191 WARN  (zkConnectionManagerCallback-253-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 434191 WARN  (zkConnectionManagerCallback-253-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 434217 INFO  (zkConnectionManagerCallback-253-thread-1-processing-n:127.0.0.1:39950_solr-EventThread) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 434217 INFO  (zkConnectionManagerCallback-253-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 434219 INFO  (zkConnectionManagerCallback-253-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 434219 INFO  (zkConnectionManagerCallback-253-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ConnectionManager zkClient Connected:true
   [junit4]   2> 434317 DEBUG (ScheduledTrigger-821-thread-4) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 434358 INFO  (OverseerCollectionConfigSetProcessor-72343775722012681-127.0.0.1:33202_solr-n_0000000000) [n:127.0.0.1:33202_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 434375 WARN  (SUITE-SystemLogListenerTest-seed#[8D83493A1500EB11]-worker-SendThread(127.0.0.1:44997)) [    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x1010447dd9f000e has expired
   [junit4]   2> 434378 WARN  (zkConnectionManagerCallback-263-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@729a8c0a name: ZooKeeperConnection Watcher:127.0.0.1:44997/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 434378 WARN  (zkConnectionManagerCallback-263-thread-1) [    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 434378 WARN  (zkConnectionManagerCallback-263-thread-1) [    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 434455 INFO  (zkConnectionManagerCallback-263-thread-1-EventThread) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 434455 INFO  (zkConnectionManagerCallback-263-thread-1) [    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 434597 INFO  (qtp1149748957-1182) [n:127.0.0.1:39950_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=3&name=.system&nrtReplicas=3&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=11425
   [junit4]   2> 434603 INFO  (zkConnectionManagerCallback-263-thread-1) [    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 434603 INFO  (zkConnectionManagerCallback-263-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient Connected:true
   [junit4]   2> 435069 INFO  (TEST-SystemLogListenerTest.test-seed#[8D83493A1500EB11]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 435343 DEBUG (ScheduledTrigger-821-thread-4) [n:127.0.0.1:33202_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 435376 INFO  (qtp1149748957-1187) [n:127.0.0.1:39950_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core..system.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=117
   [junit4]   2> 435380 INFO  (qtp1149748957-1164) [n:127.0.0.1:39950_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 435415 INFO  (qtp998215886-1176) [n:127.0.0.1:33202_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core..system.shard1.replica_n2:INDEX.sizeInBytes} status=0 QTime=1
   [junit4]   2> 435434 INFO  (qtp998215886-1171) [n:127.0.0.1:33202_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 435451 INFO  (qtp1351726253-1163) [n:127.0.0.1:35394_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core..system.shard1.replica_n3:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 435465 INFO  (qtp1351726253-1166) [n:127.0.0.1:35394_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=13
   [junit4]   2> 435465 DEBUG (qtp1149748957-1184) [n:127.0.0.1:39950_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 435469 INFO  (qtp1149748957-1184) [n:127.0.0.1:39950_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=346
   [junit4]   2> 435593 INFO  (qtp1149748957-1184) [n:127.0.0.1:39950_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core..system.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 435595 INFO  (qtp1149748957-1186) [n:127.0.0.1:39950_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 435620 INFO  (qtp998215886-1178) [n:127.0.0.1:33202

[...truncated too long message...]

teners(ScheduledTriggers.java:772)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:309)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$120/407618757@710b9f2a rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@6cf07853[Shutting down, pool size = 165, active threads = 0, queued tasks = 0, completed tasks = 1156]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:173)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.submit(SimCloudManager.java:382)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:459)
   [junit4]   2> 	... 9 more
   [junit4]   2> 1468238 WARN  (AutoscalingActionExecutor-1886-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Exception running listener {
   [junit4]   2>   "trigger":"node_lost_trigger",
   [junit4]   2>   "stage":[
   [junit4]   2>     "STARTED",
   [junit4]   2>     "ABORTED",
   [junit4]   2>     "SUCCEEDED",
   [junit4]   2>     "FAILED"],
   [junit4]   2>   "afterAction":[
   [junit4]   2>     "compute",
   [junit4]   2>     "execute"],
   [junit4]   2>   "class":"org.apache.solr.cloud.autoscaling.sim.TestLargeCluster$TestTriggerListener",
   [junit4]   2>   "beforeAction":[
   [junit4]   2>     "compute",
   [junit4]   2>     "execute"]}
   [junit4]   2> java.lang.NullPointerException
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster$TestTriggerListener.onEvent(TestLargeCluster.java:119)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:772)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:309)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1468239 WARN  (AutoscalingActionExecutor-1886-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> java.lang.Exception: Error executing action: execute for trigger event: {
   [junit4]   2>   "id":"f8844b040ef91Taokb0ycpon736s6e3tvl1396u",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":4371953246859153,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       4371953246859153,
   [junit4]   2>       4371953248906003,
   [junit4]   2>       4371953244787803,
   [junit4]   2>       4371953263905703,
   [junit4]   2>       4371953259741553,
   [junit4]   2>       4371953250924853,
   [junit4]   2>       4371953252965203,
   [junit4]   2>       4371953261890903,
   [junit4]   2>       4371953254985903,
   [junit4]   2>       4371953257005803],
   [junit4]   2>     "_enqueue_time_":4371958904557603,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10073_solr",
   [junit4]   2>       "127.0.0.1:10039_solr",
   [junit4]   2>       "127.0.0.1:10112_solr",
   [junit4]   2>       "127.0.0.1:10010_solr",
   [junit4]   2>       "127.0.0.1:10054_solr",
   [junit4]   2>       "127.0.0.1:10137_solr",
   [junit4]   2>       "127.0.0.1:10087_solr",
   [junit4]   2>       "127.0.0.1:10099_solr",
   [junit4]   2>       "127.0.0.1:10000_solr",
   [junit4]   2>       "127.0.0.1:10003_solr"]}}
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:310)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Unexpected exception while processing event: {
   [junit4]   2>   "id":"f8844b040ef91Taokb0ycpon736s6e3tvl1396u",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":4371953246859153,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       4371953246859153,
   [junit4]   2>       4371953248906003,
   [junit4]   2>       4371953244787803,
   [junit4]   2>       4371953263905703,
   [junit4]   2>       4371953259741553,
   [junit4]   2>       4371953250924853,
   [junit4]   2>       4371953252965203,
   [junit4]   2>       4371953261890903,
   [junit4]   2>       4371953254985903,
   [junit4]   2>       4371953257005803],
   [junit4]   2>     "_enqueue_time_":4371958904557603,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10073_solr",
   [junit4]   2>       "127.0.0.1:10039_solr",
   [junit4]   2>       "127.0.0.1:10112_solr",
   [junit4]   2>       "127.0.0.1:10010_solr",
   [junit4]   2>       "127.0.0.1:10054_solr",
   [junit4]   2>       "127.0.0.1:10137_solr",
   [junit4]   2>       "127.0.0.1:10087_solr",
   [junit4]   2>       "127.0.0.1:10099_solr",
   [junit4]   2>       "127.0.0.1:10000_solr",
   [junit4]   2>       "127.0.0.1:10003_solr"]}}
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:120)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
   [junit4]   2> 	... 6 more
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Unexpected exception executing operation: action=MOVEREPLICA&async=node_lost_trigger/f8844b040ef91Taokb0ycpon736s6e3tvl1396u/0&waitForFinalState=true&collection=testBasic&targetNode=127.0.0.1:10106_solr&inPlaceMove=true&replica=core_node73
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:110)
   [junit4]   2> 	... 7 more
   [junit4]   2> Caused by: java.io.IOException: java.lang.InterruptedException
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:462)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:80)
   [junit4]   2> 	... 7 more
   [junit4]   2> Caused by: java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
   [junit4]   2> 	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:460)
   [junit4]   2> 	... 8 more
   [junit4]   2> Mar 26, 2018 2:13:57 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 5 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@38f6efb8), locale=ar, timezone=AGT
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=225731576,total=503840768
   [junit4]   2> NOTE: All tests run in this JVM: [TestHighFrequencyDictionaryFactory, TestExtendedDismaxParser, TestMaxScoreQueryParser, RAMDirectoryFactoryTest, TestOmitPositions, DirectUpdateHandlerTest, TestExceedMaxTermLength, RollingRestartTest, SharedFSAutoReplicaFailoverTest, EchoParamsTest, TestExactStatsCache, SimpleFacetsTest, NodeMutatorTest, TriggerIntegrationTest, RulesTest, OutputWriterTest, TestUseDocValuesAsStored, CollectionReloadTest, ZkNodePropsTest, CdcrVersionReplicationTest, TestTolerantUpdateProcessorCloud, TokenizerChainTest, BasicDistributedZkTest, StressHdfsTest, TestFieldCacheSort, TestDocSet, TestJavabinTupleStreamParser, SolrShardReporterTest, TestHashQParserPlugin, DocValuesNotIndexedTest, TestConfigSetsAPIExclusivity, TestGroupingSearch, TestCollationField, TestCSVLoader, SolrIndexSplitterTest, TestDocTermOrds, AnalysisErrorHandlingTest, DistribJoinFromCollectionTest, ConfigSetsAPITest, CleanupOldIndexTest, TestLargeCluster]
   [junit4] Completed [148/786 (4!)] on J0 in 87.31s, 4 tests, 1 failure, 1 error <<< FAILURES!

[...truncated 51682 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 19 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/19/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ZkControllerTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [Overseer] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.Overseer  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.Overseer.start(Overseer.java:545)  at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:850)  at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)  at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)  at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)  at org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:393)  at org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:2063)  at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:331)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [Overseer]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.Overseer
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.cloud.Overseer.start(Overseer.java:545)
	at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:850)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
	at org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:393)
	at org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:2063)
	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:331)
	at java.lang.Thread.run(Thread.java:748)


	at __randomizedtesting.SeedInfo.seed([CE87C508C9594BC0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:301)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:897)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ZkControllerTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.ZkControllerTest:     1) Thread[id=2951, name=OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001, state=RUNNABLE, group=Overseer autoscaling triggers]         at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:125)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.ZkControllerTest: 
   1) Thread[id=2951, name=OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001, state=RUNNABLE, group=Overseer autoscaling triggers]
        at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:125)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([CE87C508C9594BC0]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ZkControllerTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=2951, name=OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001, state=RUNNABLE, group=Overseer autoscaling triggers]         at java.lang.Throwable.fillInStackTrace(Native Method)         at java.lang.Throwable.fillInStackTrace(Throwable.java:783)         at java.lang.Throwable.<init>(Throwable.java:310)         at java.lang.Exception.<init>(Exception.java:102)         at java.io.IOException.<init>(IOException.java:99)         at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)         at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)         at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=2951, name=OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001, state=RUNNABLE, group=Overseer autoscaling triggers]
        at java.lang.Throwable.fillInStackTrace(Native Method)
        at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
        at java.lang.Throwable.<init>(Throwable.java:310)
        at java.lang.Exception.<init>(Exception.java:102)
        at java.io.IOException.<init>(IOException.java:99)
        at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
        at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
        at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([CE87C508C9594BC0]:0)


FAILED:  org.apache.solr.logging.TestLogWatcher.testLog4jWatcher

Error Message:
expected:<50> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<50> but was:<1>
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.logging.TestLogWatcher.testLog4jWatcher(TestLogWatcher.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:251)
	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:368)
	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)




Build Log:
[...truncated 1866 lines...]
   [junit4] JVM J0: stdout was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/test/temp/junit4-J0-20180324_134402_5868456385041368903763.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] codec: FastCompressingStoredFields, pf: BlockTreeOrds, dvf: Lucene70
   [junit4] <<< JVM J0: EOF ----

[...truncated 10457 lines...]
   [junit4] Suite: org.apache.solr.cloud.ZkControllerTest
   [junit4]   2> 755212 INFO  (SUITE-ZkControllerTest-seed#[CE87C508C9594BC0]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ZkControllerTest_CE87C508C9594BC0-001/init-core-data-001
   [junit4]   2> 755212 WARN  (SUITE-ZkControllerTest-seed#[CE87C508C9594BC0]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5
   [junit4]   2> 755212 INFO  (SUITE-ZkControllerTest-seed#[CE87C508C9594BC0]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 755214 INFO  (SUITE-ZkControllerTest-seed#[CE87C508C9594BC0]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4] IGNOR/A 0.00s J1 | ZkControllerTest.testPublishAndWaitForDownStates
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028, https://issues.apache.org/jira/browse/SOLR-7736))
   [junit4]   2> 755232 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CE87C508C9594BC0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testReadConfigName
   [junit4]   2> 755232 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CE87C508C9594BC0]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 755288 INFO  (Thread-441) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 755288 INFO  (Thread-441) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 755302 ERROR (Thread-441) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 755385 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CE87C508C9594BC0]) [    ] o.a.s.c.ZkTestServer start zk server on port:33791
   [junit4]   2> 755484 INFO  (zkConnectionManagerCallback-698-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 755626 INFO  (zkConnectionManagerCallback-700-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 756191 INFO  (zkConnectionManagerCallback-702-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 756265 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CE87C508C9594BC0]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@19eb118d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 756283 INFO  (zkConnectionManagerCallback-706-thread-1-processing-n:127.0.0.1:8983_solr) [n:127.0.0.1:8983_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 756739 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CE87C508C9594BC0]) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 756754 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CE87C508C9594BC0]) [n:127.0.0.1:8983_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:8983_solr
   [junit4]   2> 756755 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CE87C508C9594BC0]) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer (id=72335646082793475-127.0.0.1:8983_solr-n_0000000000) starting
   [junit4]   2> 756902 INFO  (zkConnectionManagerCallback-711-thread-1-processing-n:127.0.0.1:8983_solr) [n:127.0.0.1:8983_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 757074 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CE87C508C9594BC0]) [n:127.0.0.1:8983_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33791/solr ready
   [junit4]   2> 757110 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CE87C508C9594BC0]) [n:127.0.0.1:8983_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:8983_solr
   [junit4]   2> 757111 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CE87C508C9594BC0]) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer (id=72335646082793475-127.0.0.1:8983_solr-n_0000000000) closing
   [junit4]   2> 757111 INFO  (OverseerStateUpdate-72335646082793475-127.0.0.1:8983_solr-n_0000000000) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:8983_solr
   [junit4]   2> 757126 WARN  (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000000) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	at java.lang.Object.wait(Object.java:502)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1409)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1328)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$setData$6(SolrZkClient.java:352)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:352)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.setData(ZkDistribStateManager.java:146)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:132)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 757130 WARN  (OverseerExitThread) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer I'm exiting, but I'm still the leader
   [junit4]   2> 757130 INFO  (zkCallback-705-thread-1-processing-n:127.0.0.1:8983_solr) [n:127.0.0.1:8983_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 757142 INFO  (zkCallback-710-thread-1-processing-n:127.0.0.1:8983_solr) [n:127.0.0.1:8983_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 757143 INFO  (OverseerExitThread) [n:127.0.0.1:8983_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:8983_solr
   [junit4]   2> 757143 INFO  (OverseerExitThread) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer (id=72335646082793475-127.0.0.1:8983_solr-n_0000000001) starting
   [junit4]   2> 757203 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757203 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757204 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757204 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757204 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757204 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757204 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757205 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757205 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757205 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757205 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757205 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757206 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757206 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757206 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757206 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757227 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757244 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757244 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757246 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757260 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757260 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757260 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757260 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757261 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757261 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757261 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757261 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757261 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757262 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757262 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757262 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757262 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757263 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757263 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757276 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757276 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757286 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757286 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757287 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757330 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757330 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757330 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757331 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 757331 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateMana

[...truncated too long message...]

ion$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 1155684 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 1155685 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 1155709 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4]   2> 1155709 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4] FAILURE 0.04s J1 | TestLogWatcher.testLog4jWatcher <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<50> but was:<1>
   [junit4]    > 	at org.apache.solr.logging.TestLogWatcher.testLog4jWatcher(TestLogWatcher.java:52)
   [junit4]   2> 1155710 ERROR (OverseerAutoScalingTriggerThread-72335646082793475-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.a.OverseerTriggerThread A ZK error has occurred
   [junit4]   2> java.io.IOException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:183)
   [junit4]   2> 	at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]   2> 	... 3 more
   [junit4] Completed [464/785 (2!)] on J1 in 0.07s, 1 test, 1 failure <<< FAILURES!

[...truncated 50476 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 18 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/18/

4 tests failed.
FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLowerTerms

Error Message:
Address already in use

Stack Trace:
java.net.BindException: Address already in use
	at __randomizedtesting.SeedInfo.seed([76AEA6D81A9CD035:F89A869BDA3B7041]:0)
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:433)
	at sun.nio.ch.Net.bind(Net.java:425)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:334)
	at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:302)
	at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
	at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:238)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.server.Server.doStart(Server.java:397)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)
	at org.apache.solr.cloud.ForceLeaderTest.bringBackOldLeaderAndSendDoc(ForceLeaderTest.java:396)
	at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLowerTerms(ForceLeaderTest.java:144)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost

Error Message:
java.util.concurrent.ExecutionException: java.io.IOException: org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{suspend-trigger={name=.scheduled_maintenance}, errorMessages=[No trigger exists with name: .scheduled_maintenance]}], 

Stack Trace:
java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{suspend-trigger={name=.scheduled_maintenance}, errorMessages=[No trigger exists with name: .scheduled_maintenance]}], 
	at __randomizedtesting.SeedInfo.seed([76AEA6D81A9CD035:C9BB68269976B5B3]:0)
	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:462)
	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:336)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.setupTest(TestLargeCluster.java:105)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:968)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{suspend-trigger={name=.scheduled_maintenance}, errorMessages=[No trigger exists with name: .scheduled_maintenance]}], 
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:460)
	... 41 more
Caused by: java.io.IOException: org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{suspend-trigger={name=.scheduled_maintenance}, errorMessages=[No trigger exists with name: .scheduled_maintenance]}], 
	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:500)
	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$2(SimCloudManager.java:459)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more
Caused by: org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{suspend-trigger={name=.scheduled_maintenance}, errorMessages=[No trigger exists with name: .scheduled_maintenance]}], 
	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.processOps(AutoScalingHandler.java:207)
	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:145)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:497)
	... 6 more


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:10006_solr should have been deleted

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:10006_solr should have been deleted
	at __randomizedtesting.SeedInfo.seed([76AEA6D81A9CD035:6E142ED414A91DDA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertFalse(Assert.java:68)
	at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration(TestTriggerIntegration.java:856)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestBlobHandler.doBlobHandlerTest

Error Message:
KeeperErrorCode = Session expired for /clusterstate.json

Stack Trace:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
	at __randomizedtesting.SeedInfo.seed([76AEA6D81A9CD035:966F848AA170A6C7]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
	at org.apache.solr.common.cloud.ZkStateReader.refreshLegacyClusterState(ZkStateReader.java:557)
	at org.apache.solr.common.cloud.ZkStateReader.forceUpdateCollection(ZkStateReader.java:363)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.updateMappingsFromZk(AbstractFullDistribZkTestBase.java:681)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.updateMappingsFromZk(AbstractFullDistribZkTestBase.java:676)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:471)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:341)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1855 lines...]
   [junit4] JVM J0: stdout was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/test/temp/junit4-J0-20180323_033715_6771986116557037081125.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] codec: Lucene70, pf: BlockTreeOrds, dvf: Memory
   [junit4] <<< JVM J0: EOF ----

[...truncated 10233 lines...]
   [junit4] Suite: org.apache.solr.handler.TestBlobHandler
   [junit4]   2> 0    INFO  (SUITE-TestBlobHandler-seed#[76AEA6D81A9CD035]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/init-core-data-001
   [junit4]   2> 778  INFO  (SUITE-TestBlobHandler-seed#[76AEA6D81A9CD035]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1033 INFO  (SUITE-TestBlobHandler-seed#[76AEA6D81A9CD035]-worker) [    ] o.e.j.u.log Logging initialized @14839ms to org.eclipse.jetty.util.log.Slf4jLog
   [junit4]   2> 1052 INFO  (SUITE-TestBlobHandler-seed#[76AEA6D81A9CD035]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1146 INFO  (SUITE-TestBlobHandler-seed#[76AEA6D81A9CD035]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1418 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1437 INFO  (Thread-1) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1438 INFO  (Thread-1) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1768 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.ZkTestServer start zk server on port:33367
   [junit4]   2> 2976 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Exception causing close of session 0x0: ZooKeeperServer not running
   [junit4]   2> 2997 ERROR (Thread-1) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 5285 INFO  (zkConnectionManagerCallback-2-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5554 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100f57f54740000, likely client has closed socket
   [junit4]   2> 5772 INFO  (zkConnectionManagerCallback-4-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5963 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 6093 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 6187 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 6296 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 6386 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 6426 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 6459 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 6479 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 6516 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 6552 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 6641 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 6702 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 12138 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T04:27:37+07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 12714 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 12714 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 12748 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 12930 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@43200246{/,null,AVAILABLE}
   [junit4]   2> 13259 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@877b9f7{HTTP/1.1,[http/1.1]}{127.0.0.1:45171}
   [junit4]   2> 13259 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.Server Started @27085ms
   [junit4]   2> 13260 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=45171, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/control-001/cores}
   [junit4]   2> 13583 ERROR (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 13583 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 13624 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 13653 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 13745 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-23T05:02:53.604Z
   [junit4]   2> 13962 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 14110 INFO  (zkConnectionManagerCallback-6-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 14205 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 14207 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/control-001/solr.xml
   [junit4]   2> 15215 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 15216 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 15391 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 17478 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33367/solr
   [junit4]   2> 17651 INFO  (zkConnectionManagerCallback-10-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 17804 INFO  (zkConnectionManagerCallback-12-thread-1-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 19637 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:45171_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 19674 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:45171_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:45171_
   [junit4]   2> 19676 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:45171_    ] o.a.s.c.Overseer Overseer (id=72327521264467972-127.0.0.1:45171_-n_0000000000) starting
   [junit4]   2> 19968 INFO  (zkConnectionManagerCallback-17-thread-1-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 19984 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:45171_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33367/solr ready
   [junit4]   2> 20342 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:45171_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45171_
   [junit4]   2> 20785 INFO  (zkCallback-16-thread-1-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 20791 INFO  (zkCallback-11-thread-1-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 26702 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:45171_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a
   [junit4]   2> 26736 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:45171_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a
   [junit4]   2> 26736 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:45171_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a
   [junit4]   2> 27274 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:45171_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/control-001/cores
   [junit4]   2> 28355 INFO  (zkConnectionManagerCallback-21-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 28483 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 28501 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33367/solr ready
   [junit4]   2> 30700 INFO  (qtp83117989-26) [n:127.0.0.1:45171_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:45171_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 30825 INFO  (OverseerThreadFactory-8-thread-1-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 31403 INFO  (qtp83117989-28) [n:127.0.0.1:45171_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 31419 INFO  (qtp83117989-28) [n:127.0.0.1:45171_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 31627 INFO  (zkCallback-11-thread-1-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 32963 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 34329 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 38671 WARN  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.WordDelimiterFilterFactory]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 39398 WARN  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.GeoHashField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 39523 WARN  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.LatLonType]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 39691 WARN  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.CurrencyField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 41150 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 42955 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 43776 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a
   [junit4]   2> 43833 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 43855 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 44130 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=35.4697265625, floorSegmentMB=0.85546875, forceMergeDeletesPctAllowed=6.024100791342029, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 45932 WARN  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 48967 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 48968 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 49041 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 49041 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 49135 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=708285363, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 49243 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@74e2739c[control_collection_shard1_replica_n1] main]
   [junit4]   2> 49334 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 49461 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 49589 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 50692 INFO  (searcherExecutor-11-thread-1-processing-n:127.0.0.1:45171_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@74e2739c[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 50729 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595703464484143104
   [junit4]   2> 50855 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 50919 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 50919 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 50919 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:45171/control_collection_shard1_replica_n1/
   [junit4]   2> 50938 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 50938 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:45171/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 50938 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 51112 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:45171/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 51148 INFO  (zkCallback-11-thread-2-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 51164 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 51238 INFO  (qtp83117989-28) [n:127.0.0.1:45171_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=19869
   [junit4]   2> 51389 INFO  (zkCallback-11-thread-2-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 51701 INFO  (qtp83117989-26) [n:127.0.0.1:45171_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 51701 INFO  (qtp83117989-26) [n:127.0.0.1:45171_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:45171_&wt=javabin&version=2} status=0 QTime=21256
   [junit4]   2> 51821 INFO  (zkConnectionManagerCallback-25-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 51826 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 51828 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33367/solr ready
   [junit4]   2> 51858 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 51862 INFO  (qtp83117989-28) [n:127.0.0.1:45171_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 51902 INFO  (OverseerCollectionConfigSetProcessor-72327521264467972-127.0.0.1:45171_-n_0000000000) [n:127.0.0.1:45171_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 51927 INFO  (OverseerThreadFactory-8-thread-2-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 51937 WARN  (OverseerThreadFactory-8-thread-2-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 52283 INFO  (qtp83117989-28) [n:127.0.0.1:45171_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 52283 INFO  (qtp83117989-28) [n:127.0.0.1:45171_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=421
   [junit4]   2> 53069 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-1-001 of type NRT
   [junit4]   2> 53070 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T04:27:37+07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 53194 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 53194 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 53195 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 53195 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7737c569{/,null,AVAILABLE}
   [junit4]   2> 53196 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5ccddeda{HTTP/1.1,[http/1.1]}{127.0.0.1:33558}
   [junit4]   2> 53196 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.Server Started @67022ms
   [junit4]   2> 53196 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=33558, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-1-001/cores}
   [junit4]   2> 53197 ERROR (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 53197 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 53197 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 53197 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 53197 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-23T05:03:33.147Z
   [junit4]   2> 53279 INFO  (zkConnectionManagerCallback-27-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 53281 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 53282 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-1-001/solr.xml
   [junit4]   2> 53313 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 53314 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 53454 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 53487 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33367/solr
   [junit4]   2> 53580 INFO  (zkConnectionManagerCallback-31-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 53742 INFO  (zkConnectionManagerCallback-33-thread-1-processing-n:127.0.0.1:33558_) [n:127.0.0.1:33558_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 53818 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:33558_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 53852 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:33558_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 53888 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:33558_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33558_
   [junit4]   2> 53925 INFO  (zkCallback-16-thread-1-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 54071 INFO  (zkCallback-24-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 54072 INFO  (OverseerCollectionConfigSetProcessor-72327521264467972-127.0.0.1:45171_-n_0000000000) [n:127.0.0.1:45171_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 54091 INFO  (zkCallback-11-thread-2-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 54144 INFO  (zkCallback-32-thread-1-processing-n:127.0.0.1:33558_) [n:127.0.0.1:33558_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 56012 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:33558_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a
   [junit4]   2> 56355 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:33558_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a
   [junit4]   2> 56355 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:33558_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a
   [junit4]   2> 56391 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:33558_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-1-001/cores
   [junit4]   2> 56517 INFO  (zkConnectionManagerCallback-38-thread-1-processing-n:127.0.0.1:33558_) [n:127.0.0.1:33558_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 56530 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:33558_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 56531 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:33558_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33367/solr ready
   [junit4]   2> 56983 INFO  (qtp1600884695-80) [n:127.0.0.1:33558_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:33558_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 57075 INFO  (OverseerThreadFactory-8-thread-3-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:33558_ for creating new replica
   [junit4]   2> 57492 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 57528 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 58759 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 59067 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema [collection1_shard2_replica_n41] Schema name=test
   [junit4]   2> 59971 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 60285 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using configuration from collection collection1, trusted=true
   [junit4]   2> 60323 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n41' (registry 'solr.core.collection1.shard2.replica_n41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a
   [junit4]   2> 60323 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 60323 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [[collection1_shard2_replica_n41] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-1-001/cores/collection1_shard2_replica_n41], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
   [junit4]   2> 60357 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=35.4697265625, floorSegmentMB=0.85546875, forceMergeDeletesPctAllowed=6.024100791342029, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 60432 WARN  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 61051 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 61088 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 61163 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 61163 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 61279 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=708285363, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 61279 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.SolrIndexSearcher Opening [Searcher@4368b744[collection1_shard2_replica_n41] main]
   [junit4]   2> 61281 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 61282 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 61315 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 61317 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595703475586465792
   [junit4]   2> 61362 INFO  (searcherExecutor-22-thread-1-processing-n:127.0.0.1:33558_ x:collection1_shard2_replica_n41 s:shard2 c:collection1) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [collection1_shard2_replica_n41] Registered new searcher Searcher@4368b744[collection1_shard2_replica_n41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 61413 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node42=0}, version=0}
   [junit4]   2> 61470 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 61470 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 61470 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33558/collection1_shard2_replica_n41/
   [junit4]   2> 61470 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 61470 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy http://127.0.0.1:33558/collection1_shard2_replica_n41/ has no replicas
   [junit4]   2> 61470 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 61500 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33558/collection1_shard2_replica_n41/ shard2
   [junit4]   2> 61589 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 61614 INFO  (qtp1600884695-83) [n:127.0.0.1:33558_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=4122
   [junit4]   2> 61634 INFO  (qtp1600884695-80) [n:127.0.0.1:33558_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:33558_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=4651
   [junit4]   2> 62359 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-2-001 of type NRT
   [junit4]   2> 62360 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T04:27:37+07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 62481 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 62481 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 62481 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 62489 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3fef217a{/,null,AVAILABLE}
   [junit4]   2> 62490 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6c0db825{HTTP/1.1,[http/1.1]}{127.0.0.1:34695}
   [junit4]   2> 62490 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.Server Started @76316ms
   [junit4]   2> 62490 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=34695, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-2-001/cores}
   [junit4]   2> 62491 ERROR (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 62491 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 62491 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 62492 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 62492 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-23T05:03:42.442Z
   [junit4]   2> 62567 INFO  (zkConnectionManagerCallback-40-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 62630 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 62630 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-2-001/solr.xml
   [junit4]   2> 62692 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 62692 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 62694 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 62721 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33367/solr
   [junit4]   2> 62736 INFO  (zkConnectionManagerCallback-44-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 62798 INFO  (zkConnectionManagerCallback-46-thread-1-processing-n:127.0.0.1:34695_) [n:127.0.0.1:34695_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 62836 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:34695_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 62872 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:34695_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 62891 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:34695_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34695_
   [junit4]   2> 62918 INFO  (zkCallback-24-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 62918 INFO  (zkCallback-37-thread-1-processing-n:127.0.0.1:33558_) [n:127.0.0.1:33558_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 62918 INFO  (zkCallback-32-thread-1-processing-n:127.0.0.1:33558_) [n:127.0.0.1:33558_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 62919 INFO  (zkCallback-16-thread-1-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 63169 INFO  (zkCallback-11-thread-1-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 63194 INFO  (zkCallback-45-thread-1-processing-n:127.0.0.1:34695_) [n:127.0.0.1:34695_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 63210 INFO  (OverseerCollectionConfigSetProcessor-72327521264467972-127.0.0.1:45171_-n_0000000000) [n:127.0.0.1:45171_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 63610 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:34695_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a
   [junit4]   2> 63683 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:34695_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a
   [junit4]   2> 63684 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:34695_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a
   [junit4]   2> 63700 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:34695_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-2-001/cores
   [junit4]   2> 63883 INFO  (zkConnectionManagerCallback-51-thread-1-processing-n:127.0.0.1:34695_) [n:127.0.0.1:34695_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 63937 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:34695_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 64010 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [n:127.0.0.1:34695_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33367/solr ready
   [junit4]   2> 64661 INFO  (qtp1600884695-81) [n:127.0.0.1:33558_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:34695_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 64830 INFO  (OverseerThreadFactory-8-thread-4-processing-n:127.0.0.1:45171_) [n:127.0.0.1:45171_    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:34695_ for creating new replica
   [junit4]   2> 64949 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 64949 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 66306 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 66536 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema [collection1_shard1_replica_n43] Schema name=test
   [junit4]   2> 67901 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 68117 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n43' using configuration from collection collection1, trusted=true
   [junit4]   2> 68118 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n43' (registry 'solr.core.collection1.shard1.replica_n43') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a
   [junit4]   2> 68118 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 68119 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [[collection1_shard1_replica_n43] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-2-001/cores/collection1_shard1_replica_n43], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-2-001/cores/collection1_shard1_replica_n43/data/]
   [junit4]   2> 68143 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=35.4697265625, floorSegmentMB=0.85546875, forceMergeDeletesPctAllowed=6.024100791342029, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 68202 WARN  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 68662 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 68662 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 68680 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 68681 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 68735 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=708285363, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 68736 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.SolrIndexSearcher Opening [Searcher@589827ed[collection1_shard1_replica_n43] main]
   [junit4]   2> 68878 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 68942 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 68943 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 68972 INFO  (searcherExecutor-33-thread-1-processing-n:127.0.0.1:34695_ x:collection1_shard1_replica_n43 s:shard1 c:collection1) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [collection1_shard1_replica_n43] Registered new searcher Searcher@589827ed[collection1_shard1_replica_n43] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 68972 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595703483613315072
   [junit4]   2> 68998 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node44=0}, version=0}
   [junit4]   2> 69095 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 69097 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 69097 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34695/collection1_shard1_replica_n43/
   [junit4]   2> 69097 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 69097 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy http://127.0.0.1:34695/collection1_shard1_replica_n43/ has no replicas
   [junit4]   2> 69097 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 69170 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:34695/collection1_shard1_replica_n43/ shard1
   [junit4]   2> 69279 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 69337 INFO  (qtp1884142125-115) [n:127.0.0.1:34695_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=4389
   [junit4]   2> 69352 INFO  (qtp1600884695-81) [n:127.0.0.1:33558_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:34695_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=4691
   [junit4]   2> 70427 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-3-001 of type NRT
   [junit4]   2> 70428 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T04:27:37+07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 70535 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 70535 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 70535 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 70536 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4797dc77{/,null,AVAILABLE}
   [junit4]   2> 70536 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@18b0e2d0{HTTP/1.1,[http/1.1]}{127.0.0.1:37139}
   [junit4]   2> 70537 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.e.j.s.Server Started @84363ms
   [junit4]   2> 70537 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=37139, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-3-001/cores}
   [junit4]   2> 70537 ERROR (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 70538 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 70538 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 70538 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 70538 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-23T05:03:50.488Z
   [junit4]   2> 70620 INFO  (zkConnectionManagerCallback-53-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 70732 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 70732 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestBlobHandler_76AEA6D81A9CD035-001/shard-3-001/solr.xml
   [junit4]   2> 70789 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 70789 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 70790 INFO  (TEST-TestBlobHandler.doBlobHandlerTest-seed#[76AEA6D81A9CD035]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3603811a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 70836 INFO  (OverseerCollectionConfigSetProcessor-72327521264467972-127.0.0.1:45171_-n_0000000000) [n:127.0.0.1:45171_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collecti

[...truncated too long message...]

junit4]   2>       4297459792205561,
   [junit4]   2>       4297459759093561,
   [junit4]   2>       4297459785569961],
   [junit4]   2>     "_enqueue_time_":4297586480060311,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10140_solr",
   [junit4]   2>       "127.0.0.1:10136_solr",
   [junit4]   2>       "127.0.0.1:10139_solr",
   [junit4]   2>       "127.0.0.1:10146_solr",
   [junit4]   2>       "127.0.0.1:10133_solr",
   [junit4]   2>       "127.0.0.1:10150_solr",
   [junit4]   2>       "127.0.0.1:10141_solr",
   [junit4]   2>       "127.0.0.1:10144_solr",
   [junit4]   2>       "127.0.0.1:10149_solr",
   [junit4]   2>       "127.0.0.1:10147_solr",
   [junit4]   2>       "127.0.0.1:10148_solr",
   [junit4]   2>       "127.0.0.1:10145_solr",
   [junit4]   2>       "127.0.0.1:10134_solr",
   [junit4]   2>       "127.0.0.1:10137_solr",
   [junit4]   2>       "127.0.0.1:10142_solr",
   [junit4]   2>       "127.0.0.1:10138_solr",
   [junit4]   2>       "127.0.0.1:10135_solr",
   [junit4]   2>       "127.0.0.1:10143_solr"]}}
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:310)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Unexpected exception while processing event: {
   [junit4]   2>   "id":"f4484533352e1T1sv7l8nlngmogbjndwbmn6yt6",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":4297459772904161,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       4297459772904161,
   [junit4]   2>       4297459777336611,
   [junit4]   2>       4297459779503111,
   [junit4]   2>       4297459775132811,
   [junit4]   2>       4297459782891061,
   [junit4]   2>       4297459761670711,
   [junit4]   2>       4297459794422111,
   [junit4]   2>       4297459796612961,
   [junit4]   2>       4297459756453361,
   [junit4]   2>       4297459798795911,
   [junit4]   2>       4297459766001511,
   [junit4]   2>       4297459787830161,
   [junit4]   2>       4297459768433661,
   [junit4]   2>       4297459770638561,
   [junit4]   2>       4297459790030461,
   [junit4]   2>       4297459792205561,
   [junit4]   2>       4297459759093561,
   [junit4]   2>       4297459785569961],
   [junit4]   2>     "_enqueue_time_":4297586480060311,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10140_solr",
   [junit4]   2>       "127.0.0.1:10136_solr",
   [junit4]   2>       "127.0.0.1:10139_solr",
   [junit4]   2>       "127.0.0.1:10146_solr",
   [junit4]   2>       "127.0.0.1:10133_solr",
   [junit4]   2>       "127.0.0.1:10150_solr",
   [junit4]   2>       "127.0.0.1:10141_solr",
   [junit4]   2>       "127.0.0.1:10144_solr",
   [junit4]   2>       "127.0.0.1:10149_solr",
   [junit4]   2>       "127.0.0.1:10147_solr",
   [junit4]   2>       "127.0.0.1:10148_solr",
   [junit4]   2>       "127.0.0.1:10145_solr",
   [junit4]   2>       "127.0.0.1:10134_solr",
   [junit4]   2>       "127.0.0.1:10137_solr",
   [junit4]   2>       "127.0.0.1:10142_solr",
   [junit4]   2>       "127.0.0.1:10138_solr",
   [junit4]   2>       "127.0.0.1:10135_solr",
   [junit4]   2>       "127.0.0.1:10143_solr"]}}
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:148)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
   [junit4]   2> 	... 6 more
   [junit4]   2> Caused by: java.lang.InterruptedException: stopping - thread was interrupted
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:100)
   [junit4]   2> 	... 7 more
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=194, maxMBSortInHeap=5.982147417535224, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@491636d6), locale=sv-SE, timezone=Asia/Kabul
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=200839808,total=531103744
   [junit4]   2> NOTE: All tests run in this JVM: [TestDistributedMap, URLClassifyProcessorTest, TestLockTree, MultiThreadedOCPTest, TestComplexPhraseLeadingWildcard, MoveReplicaHDFSFailoverTest, TestQueryTypes, TermVectorComponentTest, TestSolrConfigHandler, NoCacheHeaderTest, TestInPlaceUpdatesStandalone, TestQuerySenderListener, TestManagedSchemaAPI, LargeFieldTest, TimeZoneUtilsTest, OverseerRolesTest, RecoveryZkTest, TestEmbeddedSolrServerAdminHandler, UnloadDistributedZkTest, TestExceedMaxTermLength, AutoScalingHandlerTest, TestLegacyField, DistributedFacetSimpleRefinementLongTailTest, TestRangeQuery, SpellCheckComponentTest, ScheduledTriggerTest, CopyFieldTest, TestPolicyCloud, VMParamsZkACLAndCredentialsProvidersTest, CoreMergeIndexesAdminHandlerTest, TestTriggerIntegration, DistributedSpellCheckComponentTest, TestBadConfig, TestSchemaNameResource, TestSchemaManager, TestNonDefinedSimilarityFactory, HdfsAutoAddReplicasIntegrationTest, TestCodecSupport, TestTrieFacet, CdcrRequestHandlerTest, RegexBoostProcessorTest, DistributedFacetPivotLargeTest, MinimalSchemaTest, TestSystemCollAutoCreate, TestDistributedMissingSort, TestBlendedInfixSuggestions, AssignTest, CircularListTest, NumericFieldsTest, WordBreakSolrSpellCheckerTest, BufferStoreTest, ClusterStateUpdateTest, TestSolrDeletionPolicy2, TestInfoStreamLogging, CreateCollectionCleanupTest, SpatialHeatmapFacetsTest, DocValuesTest, HdfsDirectoryFactoryTest, TestSolrCoreSnapshots, TestConfigsApi, TestFieldCacheWithThreads, BigEndianAscendingWordSerializerTest, TestSQLHandlerNonCloud, TestDistributedStatsComponentCardinality, SimpleCollectionCreateDeleteTest, HdfsChaosMonkeySafeLeaderTest, TestMacroExpander, TestReqParamsAPI, TestDirectoryFactory, StatsReloadRaceTest, TemplateUpdateProcessorTest, TestDelegationWithHadoopAuth, TestCloudNestedDocsSort, CloneFieldUpdateProcessorFactoryTest, TestFieldCacheSort, TestConfigSets, TestSmileRequest, TestFieldSortValues, SearchHandlerTest, TestNumericTerms64, TestRealTimeGet, TestFiltering, SolrShardReporterTest, LeaderElectionTest, AsyncCallRequestStatusResponseTest, SearchHandlerTest, TestTolerantSearch, TestRequestForwarding, TestPKIAuthenticationPlugin, TestSortByMinMaxFunction, HdfsWriteToMultipleCollectionsTest, FullHLLTest, TestExactStatsCache, HdfsBasicDistributedZk2Test, TestCryptoKeys, LIROnShardRestartTest, TestRandomDVFaceting, StatelessScriptUpdateProcessorFactoryTest, CdcrBootstrapTest, DistributedFacetPivotSmallTest, TestLFUCache, InfixSuggestersTest, TestStressCloudBlindAtomicUpdates, SuggesterTSTTest, TestReloadAndDeleteDocs, DocValuesMultiTest, TestNumericRangeQuery64, TestFastWriter, TestPivotHelperCode, ReturnFieldsTest, TestFaceting, IndexSchemaRuntimeFieldTest, DisMaxRequestHandlerTest, CollectionsAPIAsyncDistributedZkTest, DistributedQueryComponentOptimizationTest, TestRawTransformer, TestReloadDeadlock, AnalysisAfterCoreReloadTest, MultiTermTest, UtilsToolTest, TestMaxScoreQueryParser, TestCoreContainer, SchemaApiFailureTest, CurrencyFieldTypeTest, PeerSyncWithIndexFingerprintCachingTest, NotRequiredUniqueKeyTest, BasicDistributedZkTest, BasicDistributedZk2Test, LeaderElectionIntegrationTest, ShardRoutingTest, TestReplicationHandler, TestDistributedGrouping, TestRecovery, TermVectorComponentDistributedTest, ZkControllerTest, TestStressReorder, DistributedTermsComponentTest, SolrCmdDistributorTest, ConvertedLegacyTest, TestLazyCores, ShowFileRequestHandlerTest, SimplePostToolTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, SuggesterFSTTest, CoreAdminHandlerTest, TestFoldingMultitermQuery, SuggesterTest, SpatialFilterTest, SuggesterWFSTTest, TestCSVLoader, PolyFieldTest, SchemaVersionSpecificBehaviorTest, TestUpdate, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, DefaultValueUpdateProcessorTest, DistanceFunctionTest, CacheHeaderTest, LukeRequestHandlerTest, TestReversedWildcardFilterFactory, TestOmitPositions, XmlUpdateRequestHandlerTest, TestValueSourceCache, FieldAnalysisRequestHandlerTest, LoggingHandlerTest, TestJmxIntegration, BinaryUpdateRequestHandlerTest, TestComponentsName, HighlighterConfigTest, UpdateParamsTest, AlternateDirectoryTest, ResponseLogComponentTest, SolrIndexConfigTest, TestConfig, TestSolrCoreProperties, DirectSolrConnectionTest, TestPhraseSuggestions, TestDFRSimilarityFactory, TestIBSimilarityFactory, DateMathParserTest, RAMDirectoryFactoryTest, ClusterStateTest, TestSolrJ, TestUtils, DistributedMLTComponentTest, CollectionPropsTest, ConnectionManagerTest, CreateRoutedAliasTest, DeleteStatusTest, DistribDocExpirationUpdateProcessorTest, DistributedQueueTest, ForceLeaderTest, LeaderFailureAfterFreshStartTest, MigrateRouteKeyTest, PeerSyncReplicationTest, SaslZkACLProviderTest, SolrCloudExampleTest, TestCloudRecovery, TestLeaderElectionZkExpiry, TestLeaderInitiatedRecoveryThread, TestPullReplicaErrorHandling, TestSSLRandomization, TestShortCircuitedRequests, TestStressLiveNodes, TestUtilizeNode, ZkFailoverTest, TestLocalFSCloudBackupRestore, AutoAddReplicasPlanActionTest, NodeAddedTriggerTest, NodeLostTriggerTest, TestComputePlanAction, TestDistribStateManager, TestGenericDistributedQueue, TestLargeCluster]
   [junit4] Completed [610/785 (4!)] on J2 in 26.36s, 4 tests, 1 error <<< FAILURES!

[...truncated 50022 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 17 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/17/

2 tests failed.
FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 8 in https://127.0.0.1:38767/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 8 in https://127.0.0.1:38767/solr
	at __randomizedtesting.SeedInfo.seed([C92F345F45EBA322:8DF4DF368BB6985]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:889)
	at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:603)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([C92F345F45EBA322:E51D43D2C92FD5A6]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:913)
	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:260)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

	request was:q=int_i:17&wt=xml
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:906)
	... 40 more




Build Log:
[...truncated 1849 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/test/temp/junit4-J0-20180322_033455_7889159593599935674289.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] codec: HighCompressionCompressingStoredFields, pf: BlockTreeOrds, dvf: Asserting
   [junit4] <<< JVM J0: EOF ----

[...truncated 11344 lines...]
   [junit4] Suite: org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest
   [junit4]   2> 1160792 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_C92F345F45EBA322-001/init-core-data-001
   [junit4]   2> 1160793 WARN  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=24 numCloses=24
   [junit4]   2> 1160793 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1160794 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1160795 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 1160795 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1160826 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1160853 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 1160977 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1161026 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1136b57a
   [junit4]   2> 1161037 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1136b57a
   [junit4]   2> 1161037 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1136b57a
   [junit4]   2> 1161039 INFO  (coreLoadExecutor-3819-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1161040 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1161074 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1161100 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1161225 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1161234 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 1161234 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1136b57a
   [junit4]   2> 1161234 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1161234 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_C92F345F45EBA322-001/init-core-data-001/]
   [junit4]   2> 1161236 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=1.9111328125, floorSegmentMB=1.603515625, forceMergeDeletesPctAllowed=28.128229811486392, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6356971818681012
   [junit4]   2> 1161301 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1161301 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1161302 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1161302 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1161303 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=20, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1161304 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6dff25e7[collection1] main]
   [junit4]   2> 1161304 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 1161312 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1161344 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1161344 INFO  (searcherExecutor-3820-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 1161344 INFO  (coreLoadExecutor-3819-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595610578901008384
   [junit4]   2> 1161345 INFO  (searcherExecutor-3820-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 1161345 INFO  (searcherExecutor-3820-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 1161345 INFO  (searcherExecutor-3820-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 1161345 INFO  (searcherExecutor-3820-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 1161345 INFO  (searcherExecutor-3820-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 1161346 INFO  (searcherExecutor-3820-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 1161346 INFO  (searcherExecutor-3820-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 1161346 INFO  (searcherExecutor-3820-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 1161347 INFO  (searcherExecutor-3820-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6dff25e7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1161348 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 1161350 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testWrongAtomicOpPassed-seed#[C92F345F45EBA322]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testWrongAtomicOpPassed
   [junit4]   2> 1161352 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testWrongAtomicOpPassed-seed#[C92F345F45EBA322]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testWrongAtomicOpPassed
   [junit4]   2> 1161355 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testNoUniqueIdPassed-seed#[C92F345F45EBA322]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNoUniqueIdPassed
   [junit4]   2> 1161355 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testNoUniqueIdPassed-seed#[C92F345F45EBA322]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNoUniqueIdPassed
   [junit4]   2> 1161358 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testBasics
   [junit4]   2> 1161362 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1595610578919882752,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1161362 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@266fe1ef commitCommandVersion:1595610578919882752
   [junit4]   2> 1161370 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@1bd8ffd3[collection1] main]
   [junit4]   2> 1161371 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1161373 INFO  (searcherExecutor-3820-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1bd8ffd3[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):c1)))}
   [junit4]   2> 1161373 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 11
   [junit4]   2> 1161376 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1161377 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:human&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1161377 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Mr&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1161378 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:20&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1161378 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=name_s:Virat&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1161379 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=multiDefault:Delhi&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1161381 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1595610578939805696,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1161381 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@266fe1ef commitCommandVersion:1595610578939805696
   [junit4]   2> 1161385 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@5bdd60ba[collection1] main]
   [junit4]   2> 1161386 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1161398 INFO  (searcherExecutor-3820-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5bdd60ba[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(8.0.0):c1)))}
   [junit4]   2> 1161398 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 17
   [junit4]   2> 1161399 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1161399 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:human&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1161400 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:animal&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1161400 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Mr&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 1161400 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Dr&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1161401 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:20&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 1161401 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:40&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1161402 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=name_s:Virat&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 1161402 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=multiDefault:Delhi&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 1161402 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[C92F345F45EBA322]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testBasics
   [junit4]   2> 1161405 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMultipleThreads
   [junit4]   2> 1161405 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1161405 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@135c1afd[collection1] realtime]
   [junit4]   2> 1161406 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 0
   [junit4]   2> 1161415 WARN  (Thread-3158) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1595610578967068672 actual=1595610578975457280Retrying with new version .... (1)
   [junit4]   2> 1161415 WARN  (Thread-3156) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1595610578967068672 actual=1595610578975457280Retrying with new version .... (1)
   [junit4]   2> 1161417 WARN  (Thread-3156) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1595610578975457280 actual=1595610578977554434Retrying with new version .... (2)
   [junit4]   2> 1161418 WARN  (Thread-3158) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1595610578975457280 actual=1595610578978603008Retrying with new version .... (2)
   [junit4]   2> 1161418 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1595610578978603010,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1161418 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@266fe1ef commitCommandVersion:1595610578978603010
   [junit4]   2> 1161422 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@2b2d7fa7[collection1] main]
   [junit4]   2> 1161423 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1161425 INFO  (searcherExecutor-3820-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2b2d7fa7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_2(8.0.0):c10/9:delGen=1)))}
   [junit4]   2> 1161425 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 7
   [junit4]   2> 1161426 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:10&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1161428 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:6vbp3wsd369772iu38q2+knj8lsj7caq5hzzk9tcz+clpcfawl168r6oe7l43q+wnxd31d0qq5o4mm6p894+i622f4idhslhytz77yea&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1161428 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=int_i:17&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 1161429 ERROR (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.SolrTestCaseJ4 REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]   2> 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2> 
   [junit4]   2> 	request was:q=int_i:17&wt=xml
   [junit4]   2> 1161429 ERROR (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.SolrTestCaseJ4 REQUEST FAILED: q=int_i:17&wt=xml:java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]   2> 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2> 
   [junit4]   2> 	request was:q=int_i:17&wt=xml
   [junit4]   2> 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:906)
   [junit4]   2> 	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:260)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 1161429 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[C92F345F45EBA322]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testMultipleThreads
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=AtomicUpdateProcessorFactoryTest -Dtests.method=testMultipleThreads -Dtests.seed=C92F345F45EBA322 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=no-NO -Dtests.timezone=Etc/GMT+8 -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.03s J0 | AtomicUpdateProcessorFactoryTest.testMultipleThreads <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C92F345F45EBA322:E51D43D2C92FD5A6]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:913)
   [junit4]    > 	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:260)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]    > 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]    > <response>
   [junit4]    > <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]    > </response>
   [junit4]    > 	request was:q=int_i:17&wt=xml
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:906)
   [junit4]    > 	... 40 more
   [junit4]   2> 1161432 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 1161432 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2043112708
   [junit4]   2> 1161432 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1161432 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@79536cc9: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@59b4057c
   [junit4]   2> 1161437 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1161437 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4c50d114: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@30857176
   [junit4]   2> 1161441 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1161441 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7aa10f41: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@744a2761
   [junit4]   2> 1161442 INFO  (coreCloseExecutor-3825-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@746e7812
   [junit4]   2> 1161442 INFO  (coreCloseExecutor-3825-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1953396754
   [junit4]   2> 1161442 INFO  (coreCloseExecutor-3825-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@40f3b94e: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@5f09badf
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_C92F345F45EBA322-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name_s=PostingsFormat(name=MockRandom), title_stringNoNorms=PostingsFormat(name=Memory), title_stemmed=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), multiDefault=PostingsFormat(name=Memory), cat=PostingsFormat(name=Memory), id=PostingsFormat(name=MockRandom), text=PostingsFormat(name=LuceneVarGapDocFreqInterval), title=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), title_lettertok=PostingsFormat(name=MockRandom)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Memory), _version_=DocValuesFormat(name=Lucene70), range_facet_i_dv=DocValuesFormat(name=Lucene70), intDvoDefault=DocValuesFormat(name=Memory), timestamp=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1965, maxMBSortInHeap=6.816819219363315, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@706c0914), locale=no-NO, timezone=Etc/GMT+8
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=289385488,total=524812288
   [junit4]   2> NOTE: All tests run in this JVM: [SearchRateTriggerTest, TestLegacyFieldReuse, ZkStateWriterTest, TestCollectionAPI, TestStressLucene, SpellCheckCollatorWithCollapseTest, TestNamedUpdateProcessors, NodeLostTriggerTest, ClassificationUpdateProcessorIntegrationTest, TestDynamicFieldResource, MetricUtilsTest, CdcrReplicationDistributedZkTest, TestJsonFacetRefinement, DeleteNodeTest, TestLuceneMatchVersion, TestCloudManagedSchema, TestSolrQueryParser, TestCloudDeleteByQuery, TestCloudPivotFacet, RequestLoggingTest, TestSolrCLIRunExample, SolrSlf4jReporterTest, TestBulkSchemaAPI, TestSolrConfigHandlerCloud, DefaultValueUpdateProcessorTest, TestIntervalFaceting, TestDistributedSearch, CurrencyRangeFacetCloudTest, SharedFSAutoReplicaFailoverTest, InfoHandlerTest, HdfsTlogReplayBufferedWhileIndexingTest, TestQueryWrapperFilter, TestCSVResponseWriter, TestLeaderElectionZkExpiry, SynonymTokenizerTest, TestMaxTokenLenTokenizer, TestSolrDeletionPolicy1, TestExpandComponent, TestShardHandlerFactory, TestExtendedDismaxParser, TestMissingGroups, UUIDUpdateProcessorFallbackTest, TestPolicyCloud, SolrMetricReporterTest, ZkFailoverTest, RequestHandlersTest, TestInitQParser, TestCorePropertiesReload, ComputePlanActionTest, TestSerializedLuceneMatchVersion, TestPullReplica, TestStandardQParsers, AliasIntegrationTest, TestCSVLoader, TestLocalFSCloudBackupRestore, TestMinMaxOnMultiValuedField, TlogReplayBufferedWhileIndexingTest, SolrPluginUtilsTest, BitVectorTest, DistributedFacetPivotWhiteBoxTest, TestOmitPositions, SolrIndexSplitterTest, TestDownShardTolerantSearch, TestRestoreCore, ChaosMonkeyNothingIsSafeTest, TestHdfsUpdateLog, DistribJoinFromCollectionTest, RegexBytesRefFilterTest, BadIndexSchemaTest, TestFastOutputStream, TestStressInPlaceUpdates, TestBlobHandler, TestOverriddenPrefixQueryForCustomFieldType, SolrRequestParserTest, TestComponentsName, EchoParamsTest, TestMacros, TestHalfAndHalfDocValues, TestIndexingPerformance, HighlighterConfigTest, TestSchemalessBufferedUpdates, SuggestComponentContextFilterQueryTest, DistributedTermsComponentTest, TestDistribIDF, HLLSerializationTest, AtomicUpdateProcessorFactoryTest]
   [junit4] Completed [333/785 (1!)] on J0 in 0.68s, 4 tests, 1 error <<< FAILURES!

[...truncated 863 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestTlogReplica
   [junit4]   2> 2441276 INFO  (SUITE-TestTlogReplica-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_C92F345F45EBA322-001/init-core-data-001
   [junit4]   2> 2441276 INFO  (SUITE-TestTlogReplica-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2441278 INFO  (SUITE-TestTlogReplica-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2441280 INFO  (SUITE-TestTlogReplica-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_C92F345F45EBA322-001/tempDir-001
   [junit4]   2> 2441280 INFO  (SUITE-TestTlogReplica-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2441280 INFO  (Thread-6289) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2441280 INFO  (Thread-6289) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2441288 ERROR (Thread-6289) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2441380 INFO  (SUITE-TestTlogReplica-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:43371
   [junit4]   2> 2441382 INFO  (zkConnectionManagerCallback-9027-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2441388 INFO  (jetty-launcher-9024-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T15:27:37-06:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2441388 INFO  (jetty-launcher-9024-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T15:27:37-06:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2441390 INFO  (jetty-launcher-9024-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2441390 INFO  (jetty-launcher-9024-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2441390 INFO  (jetty-launcher-9024-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2441390 INFO  (jetty-launcher-9024-thread-1) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 2441390 INFO  (jetty-launcher-9024-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2441390 INFO  (jetty-launcher-9024-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 2441390 INFO  (jetty-launcher-9024-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4eb36cf{/solr,null,AVAILABLE}
   [junit4]   2> 2441390 INFO  (jetty-launcher-9024-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6f1c9c99{/solr,null,AVAILABLE}
   [junit4]   2> 2441391 INFO  (jetty-launcher-9024-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@72de7c55{SSL,[ssl, http/1.1]}{127.0.0.1:54646}
   [junit4]   2> 2441391 INFO  (jetty-launcher-9024-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@184e1a91{SSL,[ssl, http/1.1]}{127.0.0.1:38767}
   [junit4]   2> 2441391 INFO  (jetty-launcher-9024-thread-1) [    ] o.e.j.s.Server Started @2443954ms
   [junit4]   2> 2441391 INFO  (jetty-launcher-9024-thread-2) [    ] o.e.j.s.Server Started @2443954ms
   [junit4]   2> 2441391 INFO  (jetty-launcher-9024-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54646}
   [junit4]   2> 2441391 INFO  (jetty-launcher-9024-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38767}
   [junit4]   2> 2441391 ERROR (jetty-launcher-9024-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2441391 ERROR (jetty-launcher-9024-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2441392 INFO  (jetty-launcher-9024-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2441392 INFO  (jetty-launcher-9024-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2441392 INFO  (jetty-launcher-9024-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2441392 INFO  (jetty-launcher-9024-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2441392 INFO  (jetty-launcher-9024-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2441392 INFO  (jetty-launcher-9024-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2441392 INFO  (jetty-launcher-9024-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-22T04:48:27.695Z
   [junit4]   2> 2441392 INFO  (jetty-launcher-9024-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-22T04:48:27.695Z
   [junit4]   2> 2441394 INFO  (zkConnectionManagerCallback-9029-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2441394 INFO  (zkConnectionManagerCallback-9031-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2441394 INFO  (jetty-launcher-9024-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2441394 INFO  (jetty-launcher-9024-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2441407 INFO  (jetty-launcher-9024-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43371/solr
   [junit4]   2> 2441407 INFO  (jetty-launcher-9024-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43371/solr
   [junit4]   2> 2441408 INFO  (zkConnectionManagerCallback-9037-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2441409 INFO  (zkConnectionManagerCallback-9039-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2441411 INFO  (zkConnectionManagerCallback-9041-thread-1-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2441411 INFO  (zkConnectionManagerCallback-9043-thread-1-processing-n:127.0.0.1:54646_solr) [n:127.0.0.1:54646_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2441538 INFO  (jetty-launcher-9024-thread-2) [n:127.0.0.1:38767_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2441538 INFO  (jetty-launcher-9024-thread-1) [n:127.0.0.1:54646_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2441539 INFO  (jetty-launcher-9024-thread-2) [n:127.0.0.1:38767_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38767_solr
   [junit4]   2> 2441539 INFO  (jetty-launcher-9024-thread-1) [n:127.0.0.1:54646_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54646_solr
   [junit4]   2> 2441540 INFO  (jetty-launcher-9024-thread-2) [n:127.0.0.1:38767_solr    ] o.a.s.c.Overseer Overseer (id=73495847761149957-127.0.0.1:38767_solr-n_0000000000) starting
   [junit4]   2> 2441546 INFO  (zkCallback-9042-thread-1-processing-n:127.0.0.1:54646_solr) [n:127.0.0.1:54646_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2441546 INFO  (zkCallback-9040-thread-1-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2441556 INFO  (zkConnectionManagerCallback-9049-thread-1-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2441557 INFO  (jetty-launcher-9024-thread-2) [n:127.0.0.1:38767_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2441558 INFO  (jetty-launcher-9024-thread-2) [n:127.0.0.1:38767_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43371/solr ready
   [junit4]   2> 2441562 INFO  (jetty-launcher-9024-thread-2) [n:127.0.0.1:38767_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38767_solr
   [junit4]   2> 2441563 INFO  (zkCallback-9042-thread-1-processing-n:127.0.0.1:54646_solr) [n:127.0.0.1:54646_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2441568 INFO  (zkCallback-9040-thread-1-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2441568 INFO  (zkCallback-9048-thread-1-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2442650 INFO  (jetty-launcher-9024-thread-1) [n:127.0.0.1:54646_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54646.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6c858903
   [junit4]   2> 2442650 INFO  (jetty-launcher-9024-thread-2) [n:127.0.0.1:38767_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38767.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6c858903
   [junit4]   2> 2442662 INFO  (jetty-launcher-9024-thread-1) [n:127.0.0.1:54646_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54646.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6c858903
   [junit4]   2> 2442662 INFO  (jetty-launcher-9024-thread-2) [n:127.0.0.1:38767_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38767.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6c858903
   [junit4]   2> 2442663 INFO  (jetty-launcher-9024-thread-1) [n:127.0.0.1:54646_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54646.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6c858903
   [junit4]   2> 2442663 INFO  (jetty-launcher-9024-thread-2) [n:127.0.0.1:38767_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38767.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6c858903
   [junit4]   2> 2442664 INFO  (jetty-launcher-9024-thread-1) [n:127.0.0.1:54646_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_C92F345F45EBA322-001/tempDir-001/node1/.
   [junit4]   2> 2442665 INFO  (jetty-launcher-9024-thread-2) [n:127.0.0.1:38767_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_C92F345F45EBA322-001/tempDir-001/node2/.
   [junit4]   2> 2442674 INFO  (zkConnectionManagerCallback-9053-thread-1-processing-n:127.0.0.1:54646_solr) [n:127.0.0.1:54646_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2442676 INFO  (jetty-launcher-9024-thread-1) [n:127.0.0.1:54646_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2442677 INFO  (jetty-launcher-9024-thread-1) [n:127.0.0.1:54646_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43371/solr ready
   [junit4]   2> 2442748 INFO  (zkConnectionManagerCallback-9055-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2442757 INFO  (zkConnectionManagerCallback-9059-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2442758 INFO  (SUITE-TestTlogReplica-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2442759 INFO  (SUITE-TestTlogReplica-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43371/solr ready
   [junit4]   2> 2442770 INFO  (SUITE-TestTlogReplica-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.c.TestTlogReplica Using legacyCloud?: false
   [junit4]   2> 2442804 INFO  (qtp358241926-30951) [n:127.0.0.1:54646_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with params val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2442806 INFO  (qtp358241926-30951) [n:127.0.0.1:54646_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 2442845 INFO  (TEST-TestTlogReplica.testAddRemoveTlogReplica-seed#[C92F345F45EBA322]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testAddRemoveTlogReplica
   [junit4]   2> 2442850 INFO  (qtp358241926-30955) [n:127.0.0.1:54646_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_add_remove_tlog_replica&nrtReplicas=0&action=CREATE&numShards=2&tlogReplicas=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2442857 INFO  (OverseerThreadFactory-9488-thread-1-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection tlog_replica_test_add_remove_tlog_replica
   [junit4]   2> 2442969 INFO  (OverseerStateUpdate-73495847761149957-127.0.0.1:38767_solr-n_0000000000) [n:127.0.0.1:38767_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_add_remove_tlog_replica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:38767/solr",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2442972 INFO  (OverseerStateUpdate-73495847761149957-127.0.0.1:38767_solr-n_0000000000) [n:127.0.0.1:38767_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_add_remove_tlog_replica",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:54646/solr",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2443178 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2&action=CREATE&numShards=2&collection=tlog_replica_test_add_remove_tlog_replica&shard=shard2&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2443179 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2443200 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1&action=CREATE&numShards=2&collection=tlog_replica_test_add_remove_tlog_replica&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2443200 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2443307 INFO  (zkCallback-9040-thread-1-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2443307 INFO  (zkCallback-9042-thread-1-processing-n:127.0.0.1:54646_solr) [n:127.0.0.1:54646_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2443307 INFO  (zkCallback-9040-thread-2-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2443308 INFO  (zkCallback-9042-thread-2-processing-n:127.0.0.1:54646_solr) [n:127.0.0.1:54646_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2444193 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2444201 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.s.IndexSchema [tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] Schema name=minimal
   [junit4]   2> 2444214 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2444214 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2' using configuration from collection tlog_replica_test_add_remove_tlog_replica, trusted=true
   [junit4]   2> 2444215 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54646.solr.core.tlog_replica_test_add_remove_tlog_replica.shard2.replica_t2' (registry 'solr.core.tlog_replica_test_add_remove_tlog_replica.shard2.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6c858903
   [junit4]   2> 2444215 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2444215 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.SolrCore [[tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_C92F345F45EBA322-001/tempDir-001/node1/tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_C92F345F45EBA322-001/tempDir-001/node1/./tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2/data/]
   [junit4]   2> 2444222 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2444231 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.s.IndexSchema [tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] Schema name=minimal
   [junit4]   2> 2444233 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2444234 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1' using configuration from collection tlog_replica_test_add_remove_tlog_replica, trusted=true
   [junit4]   2> 2444234 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38767.solr.core.tlog_replica_test_add_remove_tlog_replica.shard1.replica_t1' (registry 'solr.core.tlog_replica_test_add_remove_tlog_replica.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6c858903
   [junit4]   2> 2444234 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2444235 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.SolrCore [[tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_C92F345F45EBA322-001/tempDir-001/node2/tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_C92F345F45EBA322-001/tempDir-001/node2/./tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1/data/]
   [junit4]   2> 2444287 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2444287 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2444289 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2444289 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2444291 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@37c0baba[tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] main]
   [junit4]   2> 2444292 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2444293 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2444294 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2444295 INFO  (searcherExecutor-9493-thread-1-processing-n:127.0.0.1:54646_solr x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2 s:shard2 c:tlog_replica_test_add_remove_tlog_replica r:core_node4) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.SolrCore [tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] Registered new searcher Searcher@37c0baba[tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2444295 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595611923714408448
   [junit4]   2> 2444301 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tlog_replica_test_add_remove_tlog_replica/terms/shard2 to Terms{values={core_node4=0}, version=0}
   [junit4]   2> 2444302 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2444302 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2444304 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2444304 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2444305 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2444305 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2444305 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:54646/solr/tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2/
   [junit4]   2> 2444305 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2444305 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.SyncStrategy https://127.0.0.1:54646/solr/tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2/ has no replicas
   [junit4]   2> 2444306 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2444306 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.ZkController tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2 stopping background replication from leader
   [junit4]   2> 2444307 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@c90f743[tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] main]
   [junit4]   2> 2444308 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2444309 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2444310 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:54646/solr/tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2/ shard2
   [junit4]   2> 2444311 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2444312 INFO  (searcherExecutor-9494-thread-1-processing-n:127.0.0.1:38767_solr x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1 s:shard1 c:tlog_replica_test_add_remove_tlog_replica r:core_node3) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.SolrCore [tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] Registered new searcher Searcher@c90f743[tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2444312 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595611923732234240
   [junit4]   2> 2444317 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tlog_replica_test_add_remove_tlog_replica/terms/shard1 to Terms{values={core_node3=0}, version=0}
   [junit4]   2> 2444320 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2444320 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2444320 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:38767/solr/tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1/
   [junit4]   2> 2444320 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2444320 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.SyncStrategy https://127.0.0.1:38767/solr/tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1/ has no replicas
   [junit4]   2> 2444320 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2444321 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.ZkController tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1 stopping background replication from leader
   [junit4]   2> 2444324 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:38767/solr/tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1/ shard1
   [junit4]   2> 2444425 INFO  (zkCallback-9042-thread-2-processing-n:127.0.0.1:54646_solr) [n:127.0.0.1:54646_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2444425 INFO  (zkCallback-9042-thread-1-processing-n:127.0.0.1:54646_solr) [n:127.0.0.1:54646_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2444426 INFO  (zkCallback-9040-thread-3-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2444426 INFO  (zkCallback-9040-thread-2-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2444461 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2444463 INFO  (qtp358241926-30961) [n:127.0.0.1:54646_solr c:tlog_replica_test_add_remove_tlog_replica s:shard2 r:core_node4 x:tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_add_remove_tlog_replica_shard2_replica_t2&action=CREATE&numShards=2&collection=tlog_replica_test_add_remove_tlog_replica&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1284
   [junit4]   2> 2444475 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2444477 INFO  (qtp1011524462-30956) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_add_remove_tlog_replica_shard1_replica_t1&action=CREATE&numShards=2&collection=tlog_replica_test_add_remove_tlog_replica&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1277
   [junit4]   2> 2444481 INFO  (qtp358241926-30955) [n:127.0.0.1:54646_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2444578 INFO  (zkCallback-9040-thread-2-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2444578 INFO  (zkCallback-9040-thread-3-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2444578 INFO  (zkCallback-9042-thread-2-processing-n:127.0.0.1:54646_solr) [n:127.0.0.1:54646_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2444578 INFO  (zkCallback-9042-thread-1-processing-n:127.0.0.1:54646_solr) [n:127.0.0.1:54646_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2444858 INFO  (OverseerCollectionConfigSetProcessor-73495847761149957-127.0.0.1:38767_solr-n_0000000000) [n:127.0.0.1:38767_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2445481 INFO  (qtp358241926-30955) [n:127.0.0.1:54646_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_add_remove_tlog_replica&nrtReplicas=0&action=CREATE&numShards=2&tlogReplicas=1&wt=javabin&version=2} status=0 QTime=2631
   [junit4]   2> 2445488 INFO  (qtp1011524462-30962) [n:127.0.0.1:38767_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params action=ADDREPLICA&collection=tlog_replica_test_add_remove_tlog_replica&shard=shard1&type=TLOG and sendToOCPQueue=true
   [junit4]   2> 2445493 INFO  (OverseerThreadFactory-9488-thread-2-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:38767_solr for creating new replica
   [junit4]   2> 2445496 INFO  (OverseerStateUpdate-73495847761149957-127.0.0.1:38767_solr-n_0000000000) [n:127.0.0.1:38767_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"addreplica",
   [junit4]   2>   "collection":"tlog_replica_test_add_remove_tlog_replica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_add_remove_tlog_replica_shard1_replica_t5",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:38767/solr",
   [junit4]   2>   "node_name":"127.0.0.1:38767_solr",
   [junit4]   2>   "type":"TLOG"} 
   [junit4]   2> 2445598 INFO  (zkCallback-9040-thread-2-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2445598 INFO  (zkCallback-9040-thread-3-processing-n:127.0.0.1:38767_solr) [n:127.0.0.1:38767_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2445598 INFO  (zkCallback-9042-thread-1-processing-n:127.0.0.1:54646_solr) [n:127.0.0.1:54646_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2445598 INFO  (zkCallback-9042-thread-2-processing-n:127.0.0.1:54646_solr) [n:127.0.0.1:54646_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_remove_tlog_replica/state.json] for collection [tlog_replica_test_add_remove_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2445698 INFO  (qtp1011524462-30960) [n:127.0.0.1:38767_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf&name=tlog_replica_test_add_remove_tlog_replica_shard1_replica_t5&action=CREATE&collection=tlog_replica_test_add_remove_tlog_replica&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2445718 INFO  (qtp1011524462-30960) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node6 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2445727 INFO  (qtp1011524462-30960) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node6 x:tlog_replica_test_add_remove_tlog_replica_shard1_replica_t5] o.a.s.s.IndexSchema [tlog_replica_test_add_remove_tlog_replica_shard1_replica_t5] Schema name=minimal
   [junit4]   2> 2445730 INFO  (qtp1011524462-30960) [n:127.0.0.1:38767_solr c:tlog_replica_test_add_remove_tlog_replica s:shard1 r:core_node6 x:tlog_replica_test_add_remove_tlog_replica_

[...truncated too long message...]

unit4]   2> 2554124 INFO  (jetty-closer-9025-thread-1) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 2554125 ERROR (SUITE-TestTlogReplica-seed#[C92F345F45EBA322]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2554125 INFO  (SUITE-TestTlogReplica-seed#[C92F345F45EBA322]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43371 43371
   [junit4]   2> 2554399 INFO  (Thread-6289) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43371 43371
   [junit4]   2> 2554400 WARN  (Thread-6289) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	33	/solr/configs/conf
   [junit4]   2> 	30	/solr/collections/tlog_replica_test_create_delete/terms/shard1
   [junit4]   2> 	28	/solr/collections/tlog_replica_test_create_delete/terms/shard2
   [junit4]   2> 	27	/solr/collections/tlog_replica_test_recovery/terms/shard1
   [junit4]   2> 	19	/solr/aliases.json
   [junit4]   2> 	19	/solr/collections/tlog_replica_test_remove_leader/terms/shard1
   [junit4]   2> 	17	/solr/collections/tlog_replica_test_basic_leader_election/terms/shard1
   [junit4]   2> 	17	/solr/collections/tlog_replica_test_kill_leader/terms/shard1
   [junit4]   2> 	17	/solr/collections/tlog_replica_test_kill_tlog_replica/terms/shard1
   [junit4]   2> 	15	/solr/collections/tlog_replica_test_real_time_get/terms/shard1
   [junit4]   2> 	12	/solr/collections/tlog_replica_test_out_of_order_db_qwith_in_place_updates/terms/shard1
   [junit4]   2> 	11	/solr/collections/tlog_replica_test_only_leader_indexes/terms/shard1
   [junit4]   2> 	10	/solr/collections/tlog_replica_test_delete_by_id/terms/shard1
   [junit4]   2> 	9	/solr/security.json
   [junit4]   2> 	7	/solr/collections/tlog_replica_test_add_remove_tlog_replica/terms/shard2
   [junit4]   2> 	6	/solr/collections/tlog_replica_test_add_remove_tlog_replica/terms/shard1
   [junit4]   2> 	4	/solr/collections/tlog_replica_test_add_docs/terms/shard1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	52	/solr/collections/tlog_replica_test_create_delete/state.json
   [junit4]   2> 	39	/solr/collections/tlog_replica_test_recovery/state.json
   [junit4]   2> 	31	/solr/collections/tlog_replica_test_out_of_order_db_qwith_in_place_updates/state.json
   [junit4]   2> 	27	/solr/collections/tlog_replica_test_kill_leader/state.json
   [junit4]   2> 	26	/solr/collections/tlog_replica_test_kill_tlog_replica/state.json
   [junit4]   2> 	26	/solr/collections/tlog_replica_test_basic_leader_election/state.json
   [junit4]   2> 	26	/solr/collections/tlog_replica_test_add_remove_tlog_replica/state.json
   [junit4]   2> 	24	/solr/collections/tlog_replica_test_remove_leader/state.json
   [junit4]   2> 	19	/solr/clusterstate.json
   [junit4]   2> 	19	/solr/clusterprops.json
   [junit4]   2> 	15	/solr/collections/tlog_replica_test_only_leader_indexes/state.json
   [junit4]   2> 	15	/solr/collections/tlog_replica_test_delete_by_id/state.json
   [junit4]   2> 	13	/solr/collections/tlog_replica_test_real_time_get/state.json
   [junit4]   2> 	6	/solr/collections/tlog_replica_test_add_docs/state.json
   [junit4]   2> 	5	/solr/autoscaling.json
   [junit4]   2> 	4	/solr/collections/tlog_replica_test_recovery/leader_elect/shard1/election/73495847761149958-core_node4-n_0000000000
   [junit4]   2> 	3	/solr/overseer_elect/election/73495847761149958-127.0.0.1:54646_solr-n_0000000001
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_create_delete/leader_elect/shard1/election/73495847761149958-core_node7-n_0000000000
   [junit4]   2> 	2	/solr/overseer_elect/election/73495847761149973-127.0.0.1:38767_solr-n_0000000004
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/73495847761149958-core_node16-n_0000000001
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	19	/solr/live_nodes
   [junit4]   2> 	19	/solr/collections
   [junit4]   2> 	5	/solr/overseer/queue
   [junit4]   2> 	5	/solr/autoscaling/events/.scheduled_maintenance
   [junit4]   2> 	5	/solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 	5	/solr/overseer/queue-work
   [junit4]   2> 	5	/solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_C92F345F45EBA322-001
   [junit4]   2> Mar 22, 2018 4:50:20 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=5, maxDocsPerChunk=6, blockSize=2), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=5, blockSize=2)), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@adb1740), locale=en-US, timezone=Etc/GMT-14
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=178154368,total=508559360
   [junit4]   2> NOTE: All tests run in this JVM: [TestClusterStateProvider, SolrCloudReportersTest, AlternateDirectoryTest, TestReplicaProperties, BlockJoinFacetDistribTest, BlockDirectoryTest, DistanceUnitsTest, SystemLogListenerTest, TestPartialUpdateDeduplication, RequiredFieldsTest, HLLUtilTest, HDFSCollectionsAPITest, TestScoreJoinQPNoScore, DistribCursorPagingTest, TestReversedWildcardFilterFactory, SolrJmxReporterCloudTest, TestElisionMultitermQuery, SortByFunctionTest, TestAuthenticationFramework, TestCloudInspectUtil, TestSolrConfigHandlerConcurrent, TestSystemIdResolver, TestRawResponseWriter, PrimitiveFieldTypeTest, MetricsHandlerTest, TimeRoutedAliasUpdateProcessorTest, TestManagedResourceStorage, TestCursorMarkWithoutUniqueKey, TestQueryUtils, SolrCLIZkUtilsTest, BadComponentTest, TestClusterStateMutator, TestStressVersions, TestUniqueKeyFieldResource, TestLeaderInitiatedRecoveryThread, TestCollapseQParserPlugin, TestObjectReleaseTracker, ResponseBuilderTest, TestSlowCompositeReaderWrapper, JsonLoaderTest, TestSearchPerf, TestSolrJ, FileUtilsTest, SolrCoreMetricManagerTest, CdcrBidirectionalTest, TestCustomDocTransformer, ZkControllerTest, TestConfigSetProperties, TestSolrCoreParser, TestSimpleTextCodec, MetricTriggerTest, TestCloudSchemaless, TestPullReplicaErrorHandling, TestOrdValues, BasicZkTest, XsltUpdateRequestHandlerTest, AutoAddReplicasPlanActionTest, CloudMLTQParserTest, TestZkChroot, PrimUtilsTest, BigEndianAscendingWordDeserializerTest, ShufflingReplicaListTransformerTest, TestLRUCache, ClassificationUpdateProcessorTest, TestDistribDocBasedVersion, SolrMetricsIntegrationTest, LoggingHandlerTest, ChaosMonkeySafeLeaderWithPullReplicasTest, FieldMutatingUpdateProcessorTest, ForceLeaderTest, TestSolrQueryResponse, TestPointFields, SimplePostToolTest, TestStressLiveNodes, V2StandaloneTest, TestStressRecovery, TestAnalyzeInfixSuggestions, TestRecoveryHdfs, DistributedIntervalFacetingTest, TestFieldTypeResource, TestFoldingMultitermQuery, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestLegacyNumericRangeQueryBuilder, HighlighterTest, TestHttpShardHandlerFactory, TestSha256AuthenticationProvider, ExplicitHLLTest, ShardRoutingCustomTest, TestTestInjection, DistributedFacetPivotSmallAdvancedTest, CdcrReplicationHandlerTest, MissingSegmentRecoveryTest, TestNRTOpen, ConfigureRecoveryStrategyTest, QueryParsingTest, TestAuthorizationFramework, ConcurrentDeleteAndCreateCollectionTest, BlobRepositoryCloudTest, ReplicationFactorTest, ScriptEngineTest, SOLR749Test, OverseerModifyCollectionTest, CSVRequestHandlerTest, CursorMarkTest, DOMUtilTest, AssignBackwardCompatibilityTest, ResourceLoaderTest, TestSolrCloudWithSecureImpersonation, TestImplicitCoreProperties, SuggesterTest, TestSubQueryTransformerCrossCore, DocumentBuilderTest, QueryEqualityTest, SolrCoreTest, ChaosMonkeySafeLeaderTest, TestSkipOverseerOperations, DeleteInactiveReplicaTest, SpatialRPTFieldTypeTest, TriLevelCompositeIdRoutingTest, HdfsRecoverLeaseTest, TestRandomCollapseQParserPlugin, TestNodeAddedTrigger, DistributedFacetExistsSmallTest, ScheduledMaintenanceTriggerTest, TestSimpleQParserPlugin, ImplicitSnitchTest, DataDrivenBlockJoinTest, TestManagedSchema, UnloadDistributedZkTest, SyncSliceTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, TestRandomFaceting, ZkSolrClientTest, TestDistributedGrouping, TestFaceting, TestRecovery, TermVectorComponentDistributedTest, TestReload, TestRangeQuery, SpellCheckComponentTest, TestGroupingSearch, PeerSyncTest, DirectUpdateHandlerTest, TestIndexSearcher, SoftAutoCommitTest, TestCrossCoreJoin, TestDistributedMissingSort, TestTolerantSearch, TestLuceneIndexBackCompat, AddReplicaTest, CleanupOldIndexTest, CloudExitableDirectoryReaderTest, CollectionPropsTest, CollectionStateFormat2Test, CollectionsAPISolrJTest, DeleteShardTest, DeleteStatusTest, DistribDocExpirationUpdateProcessorTest, DocValuesNotIndexedTest, HttpPartitionTest, LegacyCloudClusterPropTest, MigrateRouteKeyTest, MoveReplicaTest, NodeMutatorTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerStatusTest, OverseerTaskQueueTest, SSLMigrationTest, TestCloudPseudoReturnFields, TestCloudSearcherWarming, TestOnReconnectListenerSupport, TestRebalanceLeaders, TestSolrCloudWithDelegationTokens, TestTlogReplica]
   [junit4] Completed [590/785 (2!)] on J1 in 113.90s, 13 tests, 1 failure <<< FAILURES!

[...truncated 50101 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 16 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/16/

1 tests failed.
FAILED:  org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([398A3F84CF8C0B10:15B8480943487D94]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:913)
	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:260)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

	request was:q=int_i:15&wt=xml
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:906)
	... 40 more




Build Log:
[...truncated 1834 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/test/temp/junit4-J1-20180321_005251_2232999257293169848199.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] codec: CheapBastard, pf: BloomFilter, dvf: Lucene70
   [junit4] <<< JVM J1: EOF ----

[...truncated 12802 lines...]
   [junit4] Suite: org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_398A3F84CF8C0B10-001/init-core-data-001
   [junit4]   2> 3341663 WARN  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=164 numCloses=164
   [junit4]   2> 3341663 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 3341664 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 3341664 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 3341665 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 3341665 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 3341698 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 3341735 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 3341855 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3341918 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a6beaff
   [junit4]   2> 3341930 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a6beaff
   [junit4]   2> 3341930 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a6beaff
   [junit4]   2> 3341932 INFO  (coreLoadExecutor-11433-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3341933 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 3341970 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 3342009 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3342147 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3342155 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 3342156 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a6beaff
   [junit4]   2> 3342156 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3342156 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_398A3F84CF8C0B10-001/init-core-data-001/]
   [junit4]   2> 3342158 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=55.95703125, floorSegmentMB=1.068359375, forceMergeDeletesPctAllowed=13.21069535271025, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7257349409094169
   [junit4]   2> 3342223 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3342224 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3342225 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3342225 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3342226 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=21.4794921875, floorSegmentMB=1.265625, forceMergeDeletesPctAllowed=29.338601023148154, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5562657653206299
   [junit4]   2> 3342226 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1ceb9a77[collection1] main]
   [junit4]   2> 3342227 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 3342235 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 3342270 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3342270 INFO  (searcherExecutor-11434-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 3342270 INFO  (coreLoadExecutor-11433-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595512078966194176
   [junit4]   2> 3342270 INFO  (searcherExecutor-11434-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 3342270 INFO  (searcherExecutor-11434-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 3342270 INFO  (searcherExecutor-11434-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 3342271 INFO  (searcherExecutor-11434-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 3342271 INFO  (searcherExecutor-11434-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 3342271 INFO  (searcherExecutor-11434-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 3342272 INFO  (searcherExecutor-11434-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 3342272 INFO  (searcherExecutor-11434-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 3342273 INFO  (searcherExecutor-11434-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1ceb9a77[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3342274 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 3342295 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testBasics
   [junit4]   2> 3342302 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1595512078999748608,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 3342302 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@103f048d commitCommandVersion:1595512078999748608
   [junit4]   2> 3342311 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@41f42be8[collection1] main]
   [junit4]   2> 3342312 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 3342315 INFO  (searcherExecutor-11434-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@41f42be8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C1)))}
   [junit4]   2> 3342315 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 12
   [junit4]   2> 3342317 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 3342318 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:human&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 3342319 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Mr&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 3342319 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:20&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 3342319 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=name_s:Virat&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 3342320 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=multiDefault:Delhi&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 3342322 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1595512079020720128,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 3342322 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@103f048d commitCommandVersion:1595512079020720128
   [junit4]   2> 3342326 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@59797982[collection1] main]
   [junit4]   2> 3342326 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 3342329 INFO  (searcherExecutor-11434-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@59797982[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(8.0.0):C1)))}
   [junit4]   2> 3342330 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 7
   [junit4]   2> 3342330 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 3342331 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:human&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 3342331 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:animal&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 3342331 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Mr&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 3342332 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Dr&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 3342332 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:20&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 3342332 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:40&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 3342333 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=name_s:Virat&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 3342333 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=multiDefault:Delhi&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 3342333 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[398A3F84CF8C0B10]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testBasics
   [junit4]   2> 3342354 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMultipleThreads
   [junit4]   2> 3342354 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 3342355 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@42b8cf9a[collection1] realtime]
   [junit4]   2> 3342355 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 0
   [junit4]   2> 3342365 WARN  (Thread-8216) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1595512079062663168 actual=1595512079062663169Retrying with new version .... (1)
   [junit4]   2> 3342366 WARN  (Thread-8213) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1595512079062663169 actual=1595512079063711744Retrying with new version .... (1)
   [junit4]   2> 3342366 WARN  (Thread-8214) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1595512079056371712 actual=1595512079063711744Retrying with new version .... (1)
   [junit4]   2> 3342366 WARN  (Thread-8215) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1595512079056371712 actual=1595512079066857473Retrying with new version .... (1)
   [junit4]   2> 3342366 WARN  (Thread-8216) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1595512079063711744 actual=1595512079066857473Retrying with new version .... (2)
   [junit4]   2> 3342367 WARN  (Thread-8217) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1595512079062663169 actual=1595512079066857474Retrying with new version .... (1)
   [junit4]   2> 3342367 WARN  (Thread-8218) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1595512079062663169 actual=1595512079066857474Retrying with new version .... (1)
   [junit4]   2> 3342367 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1595512079067906051,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 3342367 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@103f048d commitCommandVersion:1595512079067906051
   [junit4]   2> 3342371 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@4d594a57[collection1] main]
   [junit4]   2> 3342372 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 3342374 INFO  (searcherExecutor-11434-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4d594a57[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_2(8.0.0):C10/9:delGen=1)))}
   [junit4]   2> 3342374 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 7
   [junit4]   2> 3342375 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:10&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 3342376 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:tssrafi2hn5qccgffl7y+rwf52luywy83jx253m55+wk3k2hsr0n7t1yfyx34b+xsbxnemrxpc49qnz67b4+dn06qe8mvprtw6xyzmz7&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 3342376 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=int_i:15&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 3342377 ERROR (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.SolrTestCaseJ4 REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]   2> 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2> 
   [junit4]   2> 	request was:q=int_i:15&wt=xml
   [junit4]   2> 3342377 ERROR (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.SolrTestCaseJ4 REQUEST FAILED: q=int_i:15&wt=xml:java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]   2> 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2> 
   [junit4]   2> 	request was:q=int_i:15&wt=xml
   [junit4]   2> 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:906)
   [junit4]   2> 	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:260)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 3342378 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[398A3F84CF8C0B10]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testMultipleThreads
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=AtomicUpdateProcessorFactoryTest -Dtests.method=testMultipleThreads -Dtests.seed=398A3F84CF8C0B10 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=pt-PT -Dtests.timezone=NZ -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.05s J1 | AtomicUpdateProcessorFactoryTest.testMultipleThreads <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([398A3F84CF8C0B10:15B8480943487D94]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:913)
   [junit4]    > 	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:260)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]    > 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]    > <response>
   [junit4]    > <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]    > </response>
   [junit4]    > 	request was:q=int_i:15&wt=xml
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:906)
   [junit4]    > 	... 40 more
   [junit4]   2> 3342401 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testWrongAtomicOpPassed-seed#[398A3F84CF8C0B10]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testWrongAtomicOpPassed
   [junit4]   2> 3342401 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testWrongAtomicOpPassed-seed#[398A3F84CF8C0B10]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testWrongAtomicOpPassed
   [junit4]   2> 3342422 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testNoUniqueIdPassed-seed#[398A3F84CF8C0B10]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNoUniqueIdPassed
   [junit4]   2> 3342422 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testNoUniqueIdPassed-seed#[398A3F84CF8C0B10]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNoUniqueIdPassed
   [junit4]   2> 3342422 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 3342422 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1717240594
   [junit4]   2> 3342422 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 3342423 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@288ff5f3: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@ae636c2
   [junit4]   2> 3342428 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 3342428 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2fe91ad4: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@73aa8307
   [junit4]   2> 3342431 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 3342432 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[398A3F84CF8C0B10]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2613fc13: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@cb913af
   [junit4]   2> 3342432 INFO  (coreCloseExecutor-11439-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@2a7a0cd1
   [junit4]   2> 3342432 INFO  (coreCloseExecutor-11439-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=712641745
   [junit4]   2> 3342432 INFO  (coreCloseExecutor-11439-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7169e91: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@39ccc810
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_398A3F84CF8C0B10-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name_s=FSTOrd50, title_stringNoNorms=PostingsFormat(name=Memory), title_stemmed=Lucene50(blocksize=128), multiDefault=PostingsFormat(name=Memory), cat=PostingsFormat(name=Memory), id=FSTOrd50, text=PostingsFormat(name=Direct), title=Lucene50(blocksize=128), title_lettertok=FSTOrd50}, docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Asserting), range_facet_i_dv=DocValuesFormat(name=Memory), intDvoDefault=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=1652, maxMBSortInHeap=6.122489008036293, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@68d603ab), locale=pt-PT, timezone=NZ
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=163903336,total=531628032
   [junit4]   2> NOTE: All tests run in this JVM: [ChaosMonkeySafeLeaderTest, TestSimpleQParserPlugin, TestExactSharedStatsCache, BlockJoinFacetDistribTest, TestFieldTypeResource, TestXIncludeConfig, MissingSegmentRecoveryTest, ShardSplitTest, OutOfBoxZkACLAndCredentialsProvidersTest, SimplePostToolTest, TestHttpShardHandlerFactory, TestCharFilters, SampleTest, BlobRepositoryCloudTest, ExecutePlanActionTest, SoftAutoCommitTest, TestSimpleTextCodec, VersionInfoTest, SchemaVersionSpecificBehaviorTest, ScriptEngineTest, TestElisionMultitermQuery, TimeRoutedAliasUpdateProcessorTest, TestMultiWordSynonyms, TestSolrConfigHandlerConcurrent, BasicZkTest, CustomHighlightComponentTest, TestZkChroot, TestConfigSetsAPIZkFailure, TestSubQueryTransformerCrossCore, DOMUtilTest, FastVectorHighlighterTest, TestSolrIndexConfig, TestOrdValues, HdfsRecoverLeaseTest, LeaderElectionContextKeyTest, TestCursorMarkWithoutUniqueKey, SystemLogListenerTest, SolrInfoBeanTest, SolrJmxReporterCloudTest, DistanceUnitsTest, TestSolrCloudWithHadoopAuthPlugin, PrimitiveFieldTypeTest, PreAnalyzedFieldManagedSchemaCloudTest, ResponseHeaderTest, TestStressLiveNodes, BlockDirectoryTest, QueryParsingTest, PeerSyncTest, MetricsHandlerTest, ReplicationFactorTest, TestPartialUpdateDeduplication, DeleteReplicaTest, TestCloudPseudoReturnFields, AddBlockUpdateTest, ConfigureRecoveryStrategyTest, TestFastLRUCache, SolrCoreMetricManagerTest, TestReplicationHandler, TestSimDistributedQueue, TestSearchPerf, TestRawResponseWriter, XsltUpdateRequestHandlerTest, DistributedFacetPivotSmallAdvancedTest, FileUtilsTest, TestStreamBody, TestSlowCompositeReaderWrapper, TestBM25SimilarityFactory, AssignBackwardCompatibilityTest, TestClusterStateProvider, SortByFunctionTest, TestGraphMLResponseWriter, LoggingHandlerTest, OverseerModifyCollectionTest, ResponseBuilderTest, DistributedQueryComponentCustomSortTest, TestReplicaProperties, TestCloudInspectUtil, TestClusterStateMutator, V2StandaloneTest, TestRecovery, TestLRUCache, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestDistribDocBasedVersion, SolrCoreTest, FacetPivotSmallTest, ExplicitHLLTest, TestLegacyNumericRangeQueryBuilder, TriLevelCompositeIdRoutingTest, TestAnalyzeInfixSuggestions, MetricTriggerTest, TestFunctionQuery, TestSchemaSimilarityResource, RequiredFieldsTest, SuggesterTest, TestCollectionAPIs, TestCustomDocTransformer, TestConfigSetProperties, TestNRTOpen, TestInPlaceUpdatesDistrib, TermsComponentTest, ClassificationUpdateProcessorTest, SOLR749Test, DirectSolrConnectionTest, CloudExitableDirectoryReaderTest, HdfsChaosMonkeyNothingIsSafeTest, TestSimpleTrackingShardHandler, BasicFunctionalityTest, ChaosMonkeySafeLeaderWithPullReplicasTest, TestSortableTextField, TestTolerantUpdateProcessorRandomCloud, TestOnReconnectListenerSupport, SolrCLIZkUtilsTest, SpatialRPTFieldTypeTest, TestDistributedGrouping, TestClusterProperties, ActionThrottleTest, TestManagedSynonymGraphFilterFactory, HLLUtilTest, TestEmbeddedSolrServerSchemaAPI, CursorMarkTest, ScheduledMaintenanceTriggerTest, TestSolr4Spatial, TestLeaderInitiatedRecoveryThread, TestImplicitCoreProperties, QueryEqualityTest, AnalyticsMergeStrategyTest, TestObjectReleaseTracker, ConcurrentDeleteAndCreateCollectionTest, ShardRoutingCustomTest, TestLFUCache, SolrTestCaseJ4Test, TestHighlightDedupGrouping, TestTolerantSearch, TestLuceneIndexBackCompat, CleanupOldIndexTest, ConfigSetsAPITest, ConnectionManagerTest, DeleteInactiveReplicaTest, DeleteNodeTest, DeleteShardTest, DeleteStatusTest, DistribCursorPagingTest, DistribJoinFromCollectionTest, DocValuesNotIndexedTest, HealthCheckHandlerTest, HttpPartitionOnCommitTest, HttpPartitionTest, LeaderVoteWaitTimeoutTest, MoveReplicaTest, NodeMutatorTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerRolesTest, OverseerStatusTest, OverseerTaskQueueTest, RemoteQueryErrorTest, ReplaceNodeTest, RestartWhileUpdatingTest, SSLMigrationTest, SaslZkACLProviderTest, TestCloudConsistency, TestPullReplica, TestRandomRequestDistribution, TestRebalanceLeaders, TestShortCircuitedRequests, TestSolrCloudWithKerberosAlt, TestSolrCloudWithSecureImpersonation, ZkFailoverTest, AssignTest, HdfsCollectionsAPIDistributedZkTest, SimpleCollectionCreateDeleteTest, TestCollectionsAPIViaSolrCloudCluster, AutoAddReplicasPlanActionTest, HdfsAutoAddReplicasIntegrationTest, TestComputePlanAction, TestDistribStateManager, TestExecutePlanAction, TestNodeAddedTrigger, TestNodeLostTrigger, TestPolicyCloud, BaseCdcrDistributedZkTest, CdcrBidirectionalTest, CdcrReplicationDistributedZkTest, CdcrReplicationHandlerTest, CdcrRequestHandlerTest, HDFSCollectionsAPITest, HdfsBasicDistributedZk2Test, HdfsNNFailoverTest, HdfsRestartWhileUpdatingTest, HdfsSyncSliceTest, HdfsThreadLeakTest, ZkStateReaderTest, ZkStateWriterTest, ImplicitSnitchTest, DirectoryFactoryTest, ExitableDirectoryReaderTest, TestBackupRepositoryFactory, TestConfigOverlay, TestCustomStream, TestReloadAndDeleteDocs, TestSolrXml, TestSolrCloudSnapshots, TestCoreBackup, TestHdfsBackupRestoreCore, TestSQLHandler, CoreAdminRequestStatusTest, InfoHandlerTest, SecurityConfHandlerTest, SegmentsInfoRequestHandlerTest, StatsReloadRaceTest, TestConfigsApi, TestCoreAdminApis, DistributedDebugComponentTest, DistributedFacetExistsSmallTest, DistributedFacetPivotLongTailTest, DistributedFacetPivotSmallTest, DistributedFacetPivotWhiteBoxTest, ReplicaListTransformerTest, ShufflingReplicaListTransformerTest, SpatialHeatmapFacetsTest, TestTrackingShardHandlerFactory, JavabinLoaderTest, HighlighterMaxOffsetTest, TestUnifiedSolrHighlighter, WrapperMergePolicyFactoryTest, CheckHdfsIndexTest, TestCloudSchemaless, TestReloadDeadlock, TestSearcherReuse, TestSolr4Spatial2, TestSolrCoreParser, TestSolrFieldCacheBean, TestStressUserVersions, TestXmlQParser, TestXmlQParserPlugin, TestJsonFacetRefinement, TestJsonFacets, TestAuthorizationFramework, TestFieldCache, TestFieldCacheReopen, TestFieldCacheSortRandom, TestFieldCacheVsDocValues, TestNumericTerms32, DataDrivenBlockJoinTest, TestDocBasedVersionConstraints, AtomicUpdateProcessorFactoryTest]
   [junit4] Completed [771/785 (1!)] on J1 in 0.85s, 4 tests, 1 error <<< FAILURES!

[...truncated 49508 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 15 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/15/

6 tests failed.
FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:44651/solr/MoveReplicaHDFSTest_failed_coll_true, https://127.0.0.1:46482/solr/MoveReplicaHDFSTest_failed_coll_true]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:44651/solr/MoveReplicaHDFSTest_failed_coll_true, https://127.0.0.1:46482/solr/MoveReplicaHDFSTest_failed_coll_true]
	at __randomizedtesting.SeedInfo.seed([2FF0B158D51A4C19:853D62AA62C999C9]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1105)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:885)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:992)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:818)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
	at org.apache.solr.cloud.MoveReplicaTest.testFailedMove(MoveReplicaTest.java:310)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:44651/solr/MoveReplicaHDFSTest_failed_coll_true: no servers hosting shard: shard2
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:436)
	... 46 more


FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Error from server at http://127.0.0.1:33394: KeeperErrorCode = Session expired for /configs/conf1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33394: KeeperErrorCode = Session expired for /configs/conf1
	at __randomizedtesting.SeedInfo.seed([2FF0B158D51A4C19:A7A48E827BE621E1]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.BasicDistributedZkTest.createCollection(BasicDistributedZkTest.java:643)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1612)
	at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:159)
	at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:105)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.security.BasicAuthIntegrationTest.testBasicAuth

Error Message:
expected:<200> but was:<403>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<403>
	at __randomizedtesting.SeedInfo.seed([2FF0B158D51A4C19:939EC74A7149CF63]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.security.BasicAuthIntegrationTest.executeCommand(BasicAuthIntegrationTest.java:243)
	at org.apache.solr.security.BasicAuthIntegrationTest.testBasicAuth(BasicAuthIntegrationTest.java:155)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test

Error Message:
Trigger was not fired 

Stack Trace:
java.lang.AssertionError: Trigger was not fired 
	at __randomizedtesting.SeedInfo.seed([2FF0B158D51A4C19:A7A48E827BE621E1]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test(SystemLogListenerTest.java:151)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([2FF0B158D51A4C19:3C2C6D559DE3A9D]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:913)
	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:260)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

	request was:q=int_i:22&wt=xml
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:906)
	... 40 more


FAILED:  org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN

Error Message:
expected:<1.0> but was:<0.0>

Stack Trace:
java.lang.AssertionError: expected:<1.0> but was:<0.0>
	at __randomizedtesting.SeedInfo.seed([839404BD2275C075:723576ED17CE0AE7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:443)
	at org.junit.Assert.assertEquals(Assert.java:512)
	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1860 lines...]
   [junit4] JVM J0: stdout was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/test/temp/junit4-J0-20180318_173307_433906295861486679589.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] codec: SimpleText, pf: Direct, dvf: Lucene70
   [junit4] <<< JVM J0: EOF ----

[...truncated 10362 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.StressHdfsTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001/init-core-data-001
   [junit4]   2> 214676 WARN  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=7 numCloses=7
   [junit4]   2> 214676 INFO  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 214710 INFO  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 214710 INFO  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 214710 INFO  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 218749 WARN  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 226610 WARN  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 227580 INFO  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
   [junit4]   2> 227692 WARN  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 229617 INFO  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 230091 INFO  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_lucene2.us.west_apache_org_44957_hdfs____sagsjp/webapp
   [junit4]   2> 233749 INFO  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@lucene2-us-west.apache.org:44957
   [junit4]   2> 241087 WARN  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 241108 INFO  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 241148 INFO  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_44186_datanode____.ko7vhm/webapp
   [junit4]   2> 241932 INFO  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44186
   [junit4]   2> 243744 WARN  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 243746 INFO  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 243843 INFO  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_34278_datanode____.jgrcef/webapp
   [junit4]   2> 245321 INFO  (SUITE-StressHdfsTest-seed#[2FF0B158D51A4C19]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34278
   [junit4]   2> 250545 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:36857) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 250545 ERROR (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:36857) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 251554 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xd47086938e9ee: from storage DS-0cce5ecc-f684-4f14-aebb-613964254531 node DatanodeRegistration(127.0.0.1:34634, datanodeUuid=7b0f420c-c011-469a-8d25-60ca6c47da92, infoPort=40468, infoSecurePort=0, ipcPort=40134, storageInfo=lv=-56;cid=testClusterID;nsid=1969851547;c=0), blocks: 0, hasStaleStorage: true, processing time: 2 msecs
   [junit4]   2> 251555 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xd47086938e9ee: from storage DS-e4b27388-782f-4706-b11c-43589ea78ebb node DatanodeRegistration(127.0.0.1:34634, datanodeUuid=7b0f420c-c011-469a-8d25-60ca6c47da92, infoPort=40468, infoSecurePort=0, ipcPort=40134, storageInfo=lv=-56;cid=testClusterID;nsid=1969851547;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 251591 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xd47086826a9ac: from storage DS-eac02eff-8794-4505-9360-5d8aa53b0cdc node DatanodeRegistration(127.0.0.1:35546, datanodeUuid=a2de8c71-5fce-4f3c-b5c5-0c765790898e, infoPort=34454, infoSecurePort=0, ipcPort=34294, storageInfo=lv=-56;cid=testClusterID;nsid=1969851547;c=0), blocks: 0, hasStaleStorage: true, processing time: 23 msecs
   [junit4]   2> 251592 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xd47086826a9ac: from storage DS-5f137430-175e-462e-bc7f-ba33d69363b7 node DatanodeRegistration(127.0.0.1:35546, datanodeUuid=a2de8c71-5fce-4f3c-b5c5-0c765790898e, infoPort=34454, infoSecurePort=0, ipcPort=34294, storageInfo=lv=-56;cid=testClusterID;nsid=1969851547;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 252539 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 252539 INFO  (Thread-196) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 252539 INFO  (Thread-196) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 252624 ERROR (Thread-196) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 252655 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.ZkTestServer start zk server on port:36810
   [junit4]   2> 252661 INFO  (zkConnectionManagerCallback-146-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 252701 INFO  (zkConnectionManagerCallback-148-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 252869 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 252904 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 252906 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 252940 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 252942 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 252977 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 252978 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 252997 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 252998 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 252999 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 253033 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 253085 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 253235 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100dec2531a0001, likely client has closed socket
   [junit4]   2> 254391 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 254404 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 254432 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 254432 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 254434 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@59c02a6a{/,null,AVAILABLE}
   [junit4]   2> 254453 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7a81691a{HTTP/1.1,[http/1.1]}{127.0.0.1:42597}
   [junit4]   2> 254453 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.e.j.s.Server Started @259867ms
   [junit4]   2> 254453 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:36857/hdfs__lucene2-us-west.apache.org_36857__home_jenkins_jenkins-slave_workspace_Lucene-Solr-BadApples-Tests-master_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001_tempDir-002_control_data, replicaType=NRT, hostContext=/, hostPort=42597, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001/control-001/cores}
   [junit4]   2> 254454 ERROR (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 254455 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 254455 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 254455 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 254455 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-18T19:04:38.187Z
   [junit4]   2> 254579 INFO  (zkConnectionManagerCallback-150-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 254594 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 254594 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001/control-001/solr.xml
   [junit4]   2> 254644 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 254644 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 254645 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4bb94c1c, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 254794 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:36810/solr
   [junit4]   2> 254850 INFO  (zkConnectionManagerCallback-154-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 255272 INFO  (zkConnectionManagerCallback-156-thread-1-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 256196 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:42597_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 256197 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:42597_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42597_
   [junit4]   2> 256211 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:42597_    ] o.a.s.c.Overseer Overseer (id=72302520237162500-127.0.0.1:42597_-n_0000000000) starting
   [junit4]   2> 256326 INFO  (zkConnectionManagerCallback-161-thread-1-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 256369 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:42597_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:36810/solr ready
   [junit4]   2> 256474 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:42597_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42597_
   [junit4]   2> 256612 INFO  (zkCallback-160-thread-1-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 256641 INFO  (OverseerStateUpdate-72302520237162500-127.0.0.1:42597_-n_0000000000) [n:127.0.0.1:42597_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 257910 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:42597_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4bb94c1c
   [junit4]   2> 257963 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:42597_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4bb94c1c
   [junit4]   2> 257981 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:42597_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4bb94c1c
   [junit4]   2> 257983 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:42597_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001/control-001/cores
   [junit4]   2> 258158 INFO  (zkConnectionManagerCallback-165-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 258167 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 258168 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:36810/solr ready
   [junit4]   2> 258188 INFO  (qtp473416312-697) [n:127.0.0.1:42597_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42597_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 258259 INFO  (OverseerThreadFactory-151-thread-1-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 258520 INFO  (qtp473416312-700) [n:127.0.0.1:42597_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 258520 INFO  (qtp473416312-700) [n:127.0.0.1:42597_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 258711 INFO  (zkCallback-155-thread-1-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 259653 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 260014 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 269917 WARN  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]-SendThread(127.0.0.1:36810)) [n:127.0.0.1:42597_    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 11314ms for sessionid 0x100dec2531a0005
   [junit4]   2> 269917 WARN  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]-SendThread(127.0.0.1:36810)) [    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 11749ms for sessionid 0x100dec2531a0006
   [junit4]   2> 269918 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100dec2531a0005, likely client has closed socket
   [junit4]   2> 270053 WARN  (zkConnectionManagerCallback-161-thread-1-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1c97ae5a name: ZooKeeperConnection Watcher:127.0.0.1:36810/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 270053 WARN  (zkConnectionManagerCallback-161-thread-1-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 270134 WARN  (zkConnectionManagerCallback-165-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@50c8eb7e name: ZooKeeperConnection Watcher:127.0.0.1:36810/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 270134 WARN  (zkConnectionManagerCallback-165-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 270231 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 270431 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 270432 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4bb94c1c
   [junit4]   2> 270504 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:36857/solr_hdfs_home
   [junit4]   2> 270504 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 270504 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 270504 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://lucene2-us-west.apache.org:36857/solr_hdfs_home/control_collection/core_node2/data/]
   [junit4]   2> 270506 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:36857/solr_hdfs_home/control_collection/core_node2/data/snapshot_metadata
   [junit4]   2> 270632 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 270632 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 270632 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 274023 WARN  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]-SendThread(127.0.0.1:36810)) [n:127.0.0.1:42597_    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100dec2531a0005 has expired
   [junit4]   2> 274024 WARN  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]-SendThread(127.0.0.1:36810)) [    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100dec2531a0006 has expired
   [junit4]   2> 274048 WARN  (zkConnectionManagerCallback-161-thread-1-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1c97ae5a name: ZooKeeperConnection Watcher:127.0.0.1:36810/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 274048 WARN  (zkConnectionManagerCallback-161-thread-1-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 274049 WARN  (zkConnectionManagerCallback-161-thread-1-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 274065 WARN  (zkConnectionManagerCallback-165-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@50c8eb7e name: ZooKeeperConnection Watcher:127.0.0.1:36810/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 274065 WARN  (zkConnectionManagerCallback-165-thread-1) [    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 274065 WARN  (zkConnectionManagerCallback-165-thread-1) [    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 274066 INFO  (zkConnectionManagerCallback-161-thread-1-processing-n:127.0.0.1:42597_-EventThread) [n:127.0.0.1:42597_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 274100 INFO  (zkConnectionManagerCallback-161-thread-1-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 274142 INFO  (zkConnectionManagerCallback-161-thread-1-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 274142 INFO  (zkConnectionManagerCallback-161-thread-1-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.ConnectionManager zkClient Connected:true
   [junit4]   2> 274154 INFO  (zkConnectionManagerCallback-165-thread-1-EventThread) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 274154 INFO  (zkConnectionManagerCallback-165-thread-1) [    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 274157 INFO  (zkConnectionManagerCallback-165-thread-1) [    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 274157 INFO  (zkConnectionManagerCallback-165-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient Connected:true
   [junit4]   2> 278777 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 278813 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:36857/solr_hdfs_home/control_collection/core_node2/data
   [junit4]   2> 279027 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:36857/solr_hdfs_home/control_collection/core_node2/data/index
   [junit4]   2> 279064 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 279064 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 279102 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 279103 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=491477293, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 281090 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35546 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-0cce5ecc-f684-4f14-aebb-613964254531:NORMAL:127.0.0.1:34634|RBW], ReplicaUC[[DISK]DS-eac02eff-8794-4505-9360-5d8aa53b0cdc:NORMAL:127.0.0.1:35546|RBW]]} size 69
   [junit4]   2> 281090 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:34634 is added to blk_1073741825_1001 size 69
   [junit4]   2> 281744 WARN  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 282620 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 282620 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 282620 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 282781 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 282781 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 282798 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=18, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 283753 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6bbb0c4d[control_collection_shard1_replica_n1] main]
   [junit4]   2> 283755 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 283755 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 283807 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 283873 INFO  (searcherExecutor-154-thread-1-processing-n:127.0.0.1:42597_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@6bbb0c4d[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 283873 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595303429516820480
   [junit4]   2> 283882 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 283917 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 283918 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 283918 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:42597/control_collection_shard1_replica_n1/
   [junit4]   2> 283918 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 283918 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:42597/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 283918 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 283946 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:42597/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 283968 INFO  (zkCallback-155-thread-2-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 284001 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 284003 INFO  (qtp473416312-700) [n:127.0.0.1:42597_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=25491
   [junit4]   2> 284095 INFO  (qtp473416312-697) [n:127.0.0.1:42597_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 284114 INFO  (zkCallback-155-thread-1-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 284614 INFO  (OverseerCollectionConfigSetProcessor-72302520237162500-127.0.0.1:42597_-n_0000000000) [n:127.0.0.1:42597_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 285096 INFO  (qtp473416312-697) [n:127.0.0.1:42597_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42597_&wt=javabin&version=2} status=0 QTime=26908
   [junit4]   2> 285194 INFO  (zkConnectionManagerCallback-169-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 285206 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 285207 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:36810/solr ready
   [junit4]   2> 285207 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 285209 INFO  (qtp473416312-700) [n:127.0.0.1:42597_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 285244 INFO  (OverseerThreadFactory-151-thread-2-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 285336 WARN  (OverseerThreadFactory-151-thread-2-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 285454 INFO  (qtp473416312-700) [n:127.0.0.1:42597_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 285455 INFO  (qtp473416312-700) [n:127.0.0.1:42597_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=246
   [junit4]   2> 286653 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001/shard-1-001 of type TLOG
   [junit4]   2> 286654 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 286655 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 286655 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 286655 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 286655 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@69cbd424{/,null,AVAILABLE}
   [junit4]   2> 286656 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@279c38f8{HTTP/1.1,[http/1.1]}{127.0.0.1:33394}
   [junit4]   2> 286656 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.e.j.s.Server Started @292070ms
   [junit4]   2> 286656 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://lucene2-us-west.apache.org:36857/hdfs__lucene2-us-west.apache.org_36857__home_jenkins_jenkins-slave_workspace_Lucene-Solr-BadApples-Tests-master_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001_tempDir-002_jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=33394, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001/shard-1-001/cores}
   [junit4]   2> 286657 ERROR (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 286657 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 286657 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 286657 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 286658 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-18T19:05:10.389Z
   [junit4]   2> 286676 INFO  (zkConnectionManagerCallback-171-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 286678 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 286678 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001/shard-1-001/solr.xml
   [junit4]   2> 286707 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 286707 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 286708 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4bb94c1c, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 286735 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:36810/solr
   [junit4]   2> 286812 INFO  (zkConnectionManagerCallback-175-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 286890 INFO  (zkConnectionManagerCallback-177-thread-1-processing-n:127.0.0.1:33394_) [n:127.0.0.1:33394_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 286987 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:33394_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 286989 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:33394_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 286991 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:33394_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33394_
   [junit4]   2> 287019 INFO  (zkCallback-155-thread-2-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 287072 INFO  (zkCallback-176-thread-1-processing-n:127.0.0.1:33394_) [n:127.0.0.1:33394_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 287128 INFO  (zkCallback-168-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 287133 INFO  (zkCallback-160-thread-2-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 287261 INFO  (OverseerCollectionConfigSetProcessor-72302520237162500-127.0.0.1:42597_-n_0000000000) [n:127.0.0.1:42597_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 287538 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:33394_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4bb94c1c
   [junit4]   2> 287594 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:33394_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4bb94c1c
   [junit4]   2> 287594 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:33394_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4bb94c1c
   [junit4]   2> 287624 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:33394_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001/shard-1-001/cores
   [junit4]   2> 287737 INFO  (zkConnectionManagerCallback-182-thread-1-processing-n:127.0.0.1:33394_) [n:127.0.0.1:33394_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 287740 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:33394_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 287756 INFO  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]) [n:127.0.0.1:33394_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:36810/solr ready
   [junit4]   2> 287918 INFO  (qtp1321570950-771) [n:127.0.0.1:33394_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:33394_&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 288144 INFO  (OverseerThreadFactory-151-thread-3-processing-n:127.0.0.1:42597_) [n:127.0.0.1:42597_    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:33394_ for creating new replica
   [junit4]   2> 288207 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 288208 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 288352 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 288425 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.IndexSchema [collection1_shard1_replica_t21] Schema name=test
   [junit4]   2> 289003 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 289110 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t21' using configuration from collection collection1, trusted=true
   [junit4]   2> 289111 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_t21' (registry 'solr.core.collection1.shard1.replica_t21') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4bb94c1c
   [junit4]   2> 289111 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:36857/solr_hdfs_home
   [junit4]   2> 289170 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 289170 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 289171 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SolrCore [[collection1_shard1_replica_t21] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2FF0B158D51A4C19-001/shard-1-001/cores/collection1_shard1_replica_t21], dataDir=[hdfs://lucene2-us-west.apache.org:36857/solr_hdfs_home/collection1/core_node22/data/]
   [junit4]   2> 289172 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:36857/solr_hdfs_home/collection1/core_node22/data/snapshot_metadata
   [junit4]   2> 289202 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 289202 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 289269 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 289330 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:36857/solr_hdfs_home/collection1/core_node22/data
   [junit4]   2> 289455 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:36857/solr_hdfs_home/collection1/core_node22/data/index
   [junit4]   2> 289462 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 289462 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 289495 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 289580 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=491477293, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 290237 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35546 is added to blk_1073741826_1002{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-0cce5ecc-f684-4f14-aebb-613964254531:NORMAL:127.0.0.1:34634|RBW], ReplicaUC[[DISK]DS-5f137430-175e-462e-bc7f-ba33d69363b7:NORMAL:127.0.0.1:35546|RBW]]} size 69
   [junit4]   2> 290238 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:34634 is added to blk_1073741826_1002 size 69
   [junit4]   2> 290723 WARN  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 298168 WARN  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]-SendThread(127.0.0.1:36810)) [    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 7759ms for sessionid 0x100dec2531a0009
   [junit4]   2> 298184 WARN  (TEST-StressHdfsTest.test-seed#[2FF0B158D51A4C19]-SendThread(127.0.0.1:36810)) [n:127.0.0.1:33394_    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10428ms for sessionid 0x100dec2531a000d
   [junit4]   2> 298257 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100dec2531a0009, likely client has closed socket
   [junit4]   2> 298170 WARN  (zkConnectionManagerCallback-161-thread-1-processing-n:127.0.0.1:42597_-SendThread(127.0.0.1:36810)) [n:127.0.0.1:42597_    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 9926ms for sessionid 0x100dec2531a0007
   [junit4]   2> 298273 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100dec2531a000d, likely client has closed socket
   [junit4]   2> 298317 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100dec2531a0007, likely client has closed socket
   [junit4]   2> 298370 WARN  (zkConnectionManagerCallback-169-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1ef1edbb name: ZooKeeperConnection Watcher:127.0.0.1:36810/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 298370 WARN  (zkConnectionManagerCallback-169-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 298382 WARN  (zkConnectionManagerCallback-182-thread-1-processing-n:127.0.0.1:33394_) [n:127.0.0.1:33394_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3bb29e7f name: ZooKeeperConnection Watcher:127.0.0.1:36810/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 298382 WARN  (zkConnectionManagerCallback-182-thread-1-processing-n:127.0.0.1:33394_) [n:127.0.0.1:33394_    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 298390 WARN  (zkConnectionManagerCallback-161-thread-1-processing-n:127.0.0.1:42597_-EventThread) [n:127.0.0.1:42597_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1c97ae5a name: ZooKeeperConnection Watcher:127.0.0.1:36810/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 298390 WARN  (zkConnectionManagerCallback-161-thread-1-processing-n:127.0.0.1:42597_-EventThread) [n:127.0.0.1:42597_    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 298553 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 298553 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 298553 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 298671 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 298671 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 298691 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=18, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 298886 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.SolrIndexSearcher Opening [Searcher@25ccbd15[collection1_shard1_replica_t21] main]
   [junit4]   2> 298900 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 298900 INFO  (qtp1321570950-775) [n:127.0.0.1:33394_ c:collection1 s:shard1  x:collection1_shard1_repl

[...truncated too long message...]

 [junit4]   2> 5156064 INFO  (Thread-7578) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:36911 36911
   [junit4]   2> 5156065 WARN  (Thread-7578) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	9	/solr/aliases.json
   [junit4]   2> 	4	/solr/security.json
   [junit4]   2> 	4	/solr/configs/conf
   [junit4]   2> 	3	/solr/collections/authCollection/terms/shard3
   [junit4]   2> 	2	/solr/collections/authCollection/terms/shard2
   [junit4]   2> 	2	/solr/collections/authCollection/terms/shard1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	34	/solr/collections/authCollection/state.json
   [junit4]   2> 	9	/solr/clusterstate.json
   [junit4]   2> 	9	/solr/clusterprops.json
   [junit4]   2> 	3	/solr/overseer_elect/election/72302840476663816-127.0.0.1:34502_solr-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	9	/solr/live_nodes
   [junit4]   2> 	9	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.security.BasicAuthIntegrationTest_2FF0B158D51A4C19-001
   [junit4]   2> Mar 18, 2018 8:26:20 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=399, maxMBSortInHeap=5.115399558641382, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@5e6cd50e), locale=ro, timezone=Asia/Phnom_Penh
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=263260640,total=523763712
   [junit4]   2> NOTE: All tests run in this JVM: [NotRequiredUniqueKeyTest, TestPKIAuthenticationPlugin, TestFieldCacheSort, TestRangeQuery, UninvertDocValuesMergePolicyTest, CurrencyFieldTypeTest, TestReloadAndDeleteDocs, LeaderElectionTest, TestEmbeddedSolrServerAdminHandler, SuggesterTSTTest, TestCryptoKeys, SparseHLLTest, TestConfigOverlay, TestZkAclsWithHadoopAuth, AsyncCallRequestStatusResponseTest, AddSchemaFieldsUpdateProcessorFactoryTest, NoCacheHeaderTest, DateMathParserTest, TestRealTimeGet, LeaderElectionIntegrationTest, TestDFRSimilarityFactory, BadCopyFieldTest, HdfsThreadLeakTest, TestQueryTypes, TestUtilizeNode, NodeAddedTriggerTest, BlockJoinFacetRandomTest, CoreMergeIndexesAdminHandlerTest, SaslZkACLProviderTest, CloneFieldUpdateProcessorFactoryTest, AutoScalingHandlerTest, TestExactStatsCache, TestCoreDiscovery, ShowFileRequestHandlerTest, StatsReloadRaceTest, TestSuggestSpellingConverter, TestFastWriter, BasicDistributedZk2Test, TestConfigSets, TestDirectoryFactory, TestDistributedMissingSort, PingRequestHandlerTest, TestManagedStopFilterFactory, VMParamsZkACLAndCredentialsProvidersTest, HdfsAutoAddReplicasIntegrationTest, AnalysisErrorHandlingTest, TestSchemaManager, TestFieldResource, TestSolrCloudWithDelegationTokens, TestManagedSchemaAPI, MoveReplicaHDFSFailoverTest, TestComplexPhraseLeadingWildcard, TestNonDefinedSimilarityFactory, SolrCloudExampleTest, TestReloadDeadlock, TestUtils, WordBreakSolrSpellCheckerTest, SolrCmdDistributorTest, DistribDocExpirationUpdateProcessorTest, TestInPlaceUpdatesStandalone, TestReqParamsAPI, TestJettySolrRunner, BigEndianAscendingWordSerializerTest, IndexSchemaRuntimeFieldTest, PeerSyncReplicationTest, TestStressReorder, TestValueSourceCache, TestSolrCoreProperties, TestDistributedMap, OverriddenZkACLAndCredentialsProvidersTest, TestPolicyCloud, TestRandomDVFaceting, TestRTimerTree, TestLegacyField, TestHashQParserPlugin, TestAtomicUpdateErrorCases, TestFiltering, DeleteShardTest, DirectSolrSpellCheckerTest, TestUseDocValuesAsStored, LegacyCloudClusterPropTest, TestCloudNestedDocsSort, TestBlendedInfixSuggestions, TestBadConfig, NumberUtilsTest, CustomCollectionTest, TestSchemaVersionResource, ClassificationUpdateProcessorFactoryTest, ReplaceNodeNoTargetTest, TestInfoStreamLogging, TestLegacyTerms, TestShortCircuitedRequests, TestGenericDistributedQueue, TemplateUpdateProcessorTest, DeleteLastCustomShardedReplicaTest, EnumFieldTest, TestTrieFacet, DistributedFacetPivotLargeTest, LukeRequestHandlerTest, TestConfigsApi, UnloadDistributedZkTest, TestCoreContainer, CreateCollectionCleanupTest, SmileWriterTest, MultiTermTest, TestMaxScoreQueryParser, SortSpecParsingTest, ZkStateReaderTest, ConjunctionSolrSpellCheckerTest, CoreSorterTest, TestExceedMaxTermLength, TestHdfsCloudBackupRestore, SpatialFilterTest, SolrGangliaReporterTest, CdcrRequestHandlerTest, ConvertedLegacyTest, CopyFieldTest, FullHLLTest, SolrTestCaseJ4Test, DisMaxRequestHandlerTest, HighlighterMaxOffsetTest, FileBasedSpellCheckerTest, TestDefaultStatsCache, TestSolrDeletionPolicy2, StatelessScriptUpdateProcessorFactoryTest, RegexBoostProcessorTest, SyncSliceTest, OpenCloseCoreStressTest, ShardRoutingTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, TestRandomFaceting, ZkCLITest, TestDistributedSearch, TestDistributedGrouping, TestFaceting, TestRecovery, TestHashPartitioner, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, ZkControllerTest, SimpleFacetsTest, TestSolr4Spatial, StatsComponentTest, PeerSyncTest, TestLazyCores, SolrIndexSplitterTest, SimplePostToolTest, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, SpellCheckCollatorTest, SuggesterFSTTest, CoreAdminHandlerTest, SolrRequestParserTest, TestFoldingMultitermQuery, DocValuesTest, SuggesterTest, TestTrie, SuggesterWFSTTest, TestCSVLoader, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, TestUpdate, TestWordDelimiterFilterFactory, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, DistanceFunctionTest, CacheHeaderTest, TestSurroundQueryParser, PrimitiveFieldTypeTest, XmlUpdateRequestHandlerTest, TermVectorComponentTest, FastVectorHighlighterTest, IndexSchemaTest, ReturnFieldsTest, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, JsonLoaderTest, TestPartialUpdateDeduplication, CSVRequestHandlerTest, SearchHandlerTest, TestLFUCache, PreAnalyzedFieldTest, DateFieldTest, SpellingQueryConverterTest, RAMDirectoryFactoryTest, ClusterStateTest, TestSolrJ, TestDocumentBuilder, SliceStateTest, SystemInfoHandlerTest, UUIDFieldTest, FileUtilsTest, CircularListTest, DistributedMLTComponentTest, TestRTGBase, CursorPagingTest, DistributedIntervalFacetingTest, AddReplicaTest, AssignBackwardCompatibilityTest, CollectionPropsTest, ConcurrentCreateRoutedAliasTest, ConfigSetsAPITest, CreateRoutedAliasTest, DeleteInactiveReplicaTest, DistributedVersionInfoTest, LeaderFailoverAfterPartitionTest, MultiThreadedOCPTest, OverseerModifyCollectionTest, RecoveryAfterSoftCommitTest, RestartWhileUpdatingTest, SSLMigrationTest, TestAuthenticationFramework, TestCloudDeleteByQuery, TestCloudInspectUtil, TestCloudJSONFacetJoinDomain, TestCloudRecovery, TestDistribDocBasedVersion, TestDownShardTolerantSearch, TestExclusionRuleCollectionAccess, TestLeaderElectionWithEmptyReplica, TestRandomFlRTGCloud, TestRebalanceLeaders, TestTlogReplica, CollectionsAPIDistributedZkTest, TestReplicaProperties, AutoAddReplicasPlanActionTest, TriggerIntegrationTest, TestPivotHelperCode, TestTrackingShardHandlerFactory, CheckHdfsIndexTest, TestCollationFieldDocValues, TestHalfAndHalfDocValues, TestPointFields, TestFilteredDocIdSet, TestLegacyNumericRangeQueryBuilder, TestMultiWordSynonyms, TestNoOpRegenerator, TestPayloadCheckQParserPlugin, TestRandomCollapseQParserPlugin, SimpleMLTQParserTest, TestDistribIDF, BasicAuthIntegrationTest]
   [junit4] Completed [742/785 (5!)] on J2 in 18.64s, 1 test, 1 failure <<< FAILURES!

[...truncated 7613 lines...]
   [junit4] Suite: org.apache.solr.ltr.TestLTRReRankingPipeline
   [junit4]   2> 14785 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[839404BD2275C075]) [    ] o.a.s.l.TestLTRReRankingPipeline rerank 1 documents 
   [junit4]   2> 14785 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[839404BD2275C075]) [    ] o.a.s.l.TestLTRReRankingPipeline doc 0 in pos 0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTRReRankingPipeline -Dtests.method=testDifferentTopN -Dtests.seed=839404BD2275C075 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=fr-CA -Dtests.timezone=Asia/Baghdad -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.07s J2 | TestLTRReRankingPipeline.testDifferentTopN <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1.0> but was:<0.0>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([839404BD2275C075:723576ED17CE0AE7]:0)
   [junit4]    > 	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {field=FSTOrd50, id=PostingsFormat(name=LuceneVarGapFixedInterval), final-score=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{final-score=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=571, maxMBSortInHeap=6.606602565133836, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@6b1b0a3c), locale=fr-CA, timezone=Asia/Baghdad
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=189890584,total=357040128
   [junit4]   2> NOTE: All tests run in this JVM: [TestMultipleAdditiveTreesModel, TestFeatureLogging, TestLTRWithFacet, TestLTRQParserPlugin, TestFilterSolrFeature, TestLTRReRankingPipeline]
   [junit4] Completed [14/37 (1!)] on J2 in 0.16s, 3 tests, 1 failure <<< FAILURES!

[...truncated 42016 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 14 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/14/

11 tests failed.
FAILED:  org.apache.solr.cloud.TestTlogReplica.testKillLeader

Error Message:


Stack Trace:
java.lang.NullPointerException
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:269)
	at org.apache.solr.cloud.TestTlogReplica.doReplaceLeader(TestTlogReplica.java:428)
	at org.apache.solr.cloud.TestTlogReplica.testKillLeader(TestTlogReplica.java:323)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testOnlyLeaderIndexes

Error Message:
Can not find doc 2 in https://127.0.0.1:46321/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 2 in https://127.0.0.1:46321/solr
	at __randomizedtesting.SeedInfo.seed([7DFD2B6DF7CB6300:61FC56E0826E1D93]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:889)
	at org.apache.solr.cloud.TestTlogReplica.testOnlyLeaderIndexes(TestTlogReplica.java:487)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testOutOfOrderDBQWithInPlaceUpdates

Error Message:
Error from server at https://127.0.0.1:46321/solr: Request says it is coming from leader, but we are the leader

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:46321/solr: Request says it is coming from leader, but we are the leader
	at __randomizedtesting.SeedInfo.seed([7DFD2B6DF7CB6300:FB3CD380A89AB5E0]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.cloud.TestTlogReplica.testOutOfOrderDBQWithInPlaceUpdates(TestTlogReplica.java:676)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ZkControllerTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [Overseer] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.Overseer  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.Overseer.start(Overseer.java:545)  at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:850)  at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)  at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)  at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)  at org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:393)  at org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:2063)  at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:331)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [Overseer]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.Overseer
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.cloud.Overseer.start(Overseer.java:545)
	at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:850)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
	at org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:393)
	at org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:2063)
	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:331)
	at java.lang.Thread.run(Thread.java:748)


	at __randomizedtesting.SeedInfo.seed([7DFD2B6DF7CB6300]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:301)
	at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:897)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ZkControllerTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.ZkControllerTest:     1) Thread[id=15232, name=OverseerAutoScalingTriggerThread-72298237203054595-127.0.0.1:8983_solr-n_0000000001, state=RUNNABLE, group=Overseer autoscaling triggers]         at java.lang.Object.notifyAll(Native Method)         at org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:652)         at org.apache.zookeeper.ClientCnxn.conLossPacket(ClientCnxn.java:674)         at org.apache.zookeeper.ClientCnxn.queuePacket(ClientCnxn.java:1452)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1405)         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1213)         at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)         at org.apache.solr.common.cloud.SolrZkClient$$Lambda$217/246968454.execute(Unknown Source)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)         at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)         at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)         at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.ZkControllerTest: 
   1) Thread[id=15232, name=OverseerAutoScalingTriggerThread-72298237203054595-127.0.0.1:8983_solr-n_0000000001, state=RUNNABLE, group=Overseer autoscaling triggers]
        at java.lang.Object.notifyAll(Native Method)
        at org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:652)
        at org.apache.zookeeper.ClientCnxn.conLossPacket(ClientCnxn.java:674)
        at org.apache.zookeeper.ClientCnxn.queuePacket(ClientCnxn.java:1452)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1405)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1213)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
        at org.apache.solr.common.cloud.SolrZkClient$$Lambda$217/246968454.execute(Unknown Source)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
        at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
        at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
        at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([7DFD2B6DF7CB6300]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ZkControllerTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=15232, name=OverseerAutoScalingTriggerThread-72298237203054595-127.0.0.1:8983_solr-n_0000000001, state=RUNNABLE, group=Overseer autoscaling triggers]         at java.lang.Object.notifyAll(Native Method)         at org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:652)         at org.apache.zookeeper.ClientCnxn.conLossPacket(ClientCnxn.java:674)         at org.apache.zookeeper.ClientCnxn.queuePacket(ClientCnxn.java:1452)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1405)         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1213)         at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)         at org.apache.solr.common.cloud.SolrZkClient$$Lambda$217/246968454.execute(Unknown Source)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)         at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)         at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)         at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=15232, name=OverseerAutoScalingTriggerThread-72298237203054595-127.0.0.1:8983_solr-n_0000000001, state=RUNNABLE, group=Overseer autoscaling triggers]
        at java.lang.Object.notifyAll(Native Method)
        at org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:652)
        at org.apache.zookeeper.ClientCnxn.conLossPacket(ClientCnxn.java:674)
        at org.apache.zookeeper.ClientCnxn.queuePacket(ClientCnxn.java:1452)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1405)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1213)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
        at org.apache.solr.common.cloud.SolrZkClient$$Lambda$217/246968454.execute(Unknown Source)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
        at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
        at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
        at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([7DFD2B6DF7CB6300]:0)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue

Error Message:
action wasn't interrupted

Stack Trace:
java.lang.AssertionError: action wasn't interrupted
	at __randomizedtesting.SeedInfo.seed([7DFD2B6DF7CB6300:B44869C3FEACA5F5]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue(TriggerIntegrationTest.java:752)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
	at __randomizedtesting.SeedInfo.seed([7DFD2B6DF7CB6300:C2E8E59374210686]:0)
	at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:108)
	at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.doTestNodeLost(TestLargeCluster.java:479)
	at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost(TestLargeCluster.java:320)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testBasic

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
	at __randomizedtesting.SeedInfo.seed([7DFD2B6DF7CB6300:D60736782817E52E]:0)
	at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:108)
	at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testBasic(TestLargeCluster.java:195)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testSearchRate

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
	at __randomizedtesting.SeedInfo.seed([7DFD2B6DF7CB6300:20B535E4380DC54F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testSearchRate(TestLargeCluster.java:576)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN

Error Message:
expected:<1.0> but was:<0.0>

Stack Trace:
java.lang.AssertionError: expected:<1.0> but was:<0.0>
	at __randomizedtesting.SeedInfo.seed([2025C8687D18E4BE:D184BA3848A32E2C]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:443)
	at org.junit.Assert.assertEquals(Assert.java:512)
	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1854 lines...]
   [junit4] JVM J0: stdout was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/test/temp/junit4-J0-20180317_224121_4496902511495461447057.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] codec: SimpleText, pf: FSTOrd50, dvf: Lucene70
   [junit4] <<< JVM J0: EOF ----

[...truncated 10281 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestLargeCluster
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestLargeCluster_7DFD2B6DF7CB6300-001/init-core-data-001
   [junit4]   2> 169645 WARN  (SUITE-TestLargeCluster-seed#[7DFD2B6DF7CB6300]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=6 numCloses=6
   [junit4]   2> 169658 INFO  (SUITE-TestLargeCluster-seed#[7DFD2B6DF7CB6300]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 169660 INFO  (SUITE-TestLargeCluster-seed#[7DFD2B6DF7CB6300]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 169660 INFO  (SUITE-TestLargeCluster-seed#[7DFD2B6DF7CB6300]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 169842 INFO  (TEST-TestLargeCluster.testAddNode-seed#[7DFD2B6DF7CB6300]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testAddNode
   [junit4]   2> 169843 INFO  (TEST-TestLargeCluster.testAddNode-seed#[7DFD2B6DF7CB6300]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 169843 DEBUG (TEST-TestLargeCluster.testAddNode-seed#[7DFD2B6DF7CB6300]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 169861 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 169862 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 169862 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10004_solr, 127.0.0.1:10018_solr, 127.0.0.1:10049_solr, 127.0.0.1:10020_solr, 127.0.0.1:10065_solr, 127.0.0.1:10021_solr, 127.0.0.1:10005_solr, 127.0.0.1:10066_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10019_solr, 127.0.0.1:10096_solr, 127.0.0.1:10022_solr, 127.0.0.1:10033_solr, 127.0.0.1:10017_solr, 127.0.0.1:10047_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10003_solr, 127.0.0.1:10001_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10071_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10055_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10013_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10009_solr, 127.0.0.1:10000_solr, 127.0.0.1:10030_solr, 127.0.0.1:10014_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10015_solr, 127.0.0.1:10069_solr, 127.0.0.1:10062_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10002_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10037_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr]
   [junit4]   2> 169862 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 169862 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 169844 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 169879 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 169918 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 170044 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10004_solr, 127.0.0.1:10018_solr, 127.0.0.1:10049_solr, 127.0.0.1:10020_solr, 127.0.0.1:10065_solr, 127.0.0.1:10021_solr, 127.0.0.1:10005_solr, 127.0.0.1:10066_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10019_solr, 127.0.0.1:10096_solr, 127.0.0.1:10022_solr, 127.0.0.1:10033_solr, 127.0.0.1:10017_solr, 127.0.0.1:10047_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10003_solr, 127.0.0.1:10001_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10071_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10055_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10013_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10009_solr, 127.0.0.1:10000_solr, 127.0.0.1:10030_solr, 127.0.0.1:10014_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10015_solr, 127.0.0.1:10069_solr, 127.0.0.1:10062_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10002_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10037_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr]
   [junit4]   2> 170047 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 1
   [junit4]   2> 170047 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 170047 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 170047 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 170007 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170243 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170280 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170332 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170368 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170388 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170408 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170426 DEBUG (simCloudManagerPool-205-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 170426 DEBUG (simCloudManagerPool-205-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 170427 DEBUG (simCloudManagerPool-205-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10004_solr, 127.0.0.1:10018_solr, 127.0.0.1:10049_solr, 127.0.0.1:10020_solr, 127.0.0.1:10065_solr, 127.0.0.1:10021_solr, 127.0.0.1:10005_solr, 127.0.0.1:10066_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10019_solr, 127.0.0.1:10096_solr, 127.0.0.1:10022_solr, 127.0.0.1:10033_solr, 127.0.0.1:10017_solr, 127.0.0.1:10047_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10003_solr, 127.0.0.1:10001_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10071_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10055_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10013_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10009_solr, 127.0.0.1:10000_solr, 127.0.0.1:10030_solr, 127.0.0.1:10014_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10015_solr, 127.0.0.1:10069_solr, 127.0.0.1:10062_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10002_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10037_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr]
   [junit4]   2> 170427 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 170428 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 170428 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 170438 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170459 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170462 DEBUG (simCloudManagerPool-205-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 170474 DEBUG (simCloudManagerPool-205-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 170475 DEBUG (simCloudManagerPool-205-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10004_solr, 127.0.0.1:10018_solr, 127.0.0.1:10049_solr, 127.0.0.1:10020_solr, 127.0.0.1:10065_solr, 127.0.0.1:10021_solr, 127.0.0.1:10005_solr, 127.0.0.1:10066_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10019_solr, 127.0.0.1:10096_solr, 127.0.0.1:10022_solr, 127.0.0.1:10033_solr, 127.0.0.1:10017_solr, 127.0.0.1:10047_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10003_solr, 127.0.0.1:10001_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10071_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10055_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10013_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10009_solr, 127.0.0.1:10000_solr, 127.0.0.1:10030_solr, 127.0.0.1:10014_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10015_solr, 127.0.0.1:10069_solr, 127.0.0.1:10062_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10002_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10037_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr]
   [junit4]   2> 170477 DEBUG (simCloudManagerPool-205-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10004_solr, 127.0.0.1:10018_solr, 127.0.0.1:10049_solr, 127.0.0.1:10020_solr, 127.0.0.1:10065_solr, 127.0.0.1:10021_solr, 127.0.0.1:10005_solr, 127.0.0.1:10066_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10019_solr, 127.0.0.1:10096_solr, 127.0.0.1:10022_solr, 127.0.0.1:10033_solr, 127.0.0.1:10017_solr, 127.0.0.1:10047_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10003_solr, 127.0.0.1:10001_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10071_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10055_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10013_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10009_solr, 127.0.0.1:10000_solr, 127.0.0.1:10030_solr, 127.0.0.1:10014_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10015_solr, 127.0.0.1:10069_solr, 127.0.0.1:10062_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10002_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10037_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr]
   [junit4]   2> 170477 DEBUG (simCloudManagerPool-205-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=5, actions=[{name=compute, class=org.apache.solr.cloud.autoscaling.ComputePlanAction}, {name=execute, class=org.apache.solr.cloud.autoscaling.ExecutePlanAction}, {name=test, class=org.apache.solr.cloud.autoscaling.sim.TestLargeCluster$TestTriggerAction}], enabled=true}
   [junit4]   2> 170483 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 170484 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 170494 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170584 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170495 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 170584 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 170665 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170702 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170734 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170767 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170787 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170824 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170860 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170880 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170913 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170933 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 170970 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171006 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171043 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171079 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171116 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171152 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171189 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171225 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171262 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171298 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171335 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171371 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171448 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 171449 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 171449 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171664 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 171665 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 171665 DEBUG (ScheduledTrigger-206-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171696 DEBUG (ScheduledTrigger-206-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171696 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 171698 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 171718 DEBUG (ScheduledTrigger-206-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171718 DEBUG (ScheduledTrigger-206-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 171718 DEBUG (ScheduledTrigger-206-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 171739 DEBUG (ScheduledTrigger-206-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171739 DEBUG (ScheduledTrigger-206-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 171739 DEBUG (ScheduledTrigger-206-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 171773 DEBUG (ScheduledTrigger-206-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171773 DEBUG (ScheduledTrigger-206-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 171773 DEBUG (ScheduledTrigger-206-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 171806 DEBUG (ScheduledTrigger-206-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171814 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 171814 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 171838 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 171838 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 171838 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171875 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 171875 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 171875 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171927 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 171928 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 171928 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171948 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 171948 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 171948 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 171984 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 171984 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 171985 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172021 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172021 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172021 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172057 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172058 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172058 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172094 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172094 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172094 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172130 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172131 DEBUG (ScheduledTrigger-206-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172131 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172293 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172293 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172293 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172329 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172330 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172330 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172366 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172366 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172366 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172403 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172403 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172403 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172439 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172439 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172439 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172476 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172476 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172476 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172512 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172512 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172512 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172549 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172549 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172549 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172585 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172585 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172586 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172623 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172623 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172623 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172658 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172658 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172658 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172695 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172695 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172695 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172731 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172732 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172732 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172768 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172768 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172768 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172804 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172804 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172805 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172841 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172841 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172841 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172877 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172878 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172878 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172914 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172914 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172914 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172951 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172951 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172951 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 172987 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 172987 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 172987 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173024 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173024 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173024 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173076 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173076 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173077 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173113 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173113 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173113 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173149 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173149 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173150 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173186 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173186 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173186 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173222 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173223 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173223 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173259 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173259 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173259 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173328 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173328 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173329 DEBUG (ScheduledTrigger-206-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173349 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173349 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173349 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173385 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173385 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173385 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173421 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173422 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173422 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173458 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173458 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173458 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173494 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173494 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173495 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173531 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173531 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173531 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173567 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173568 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173568 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 173604 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 173604 DEBUG (ScheduledTrigger-206-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 100
   [junit4]   2> 173604 DEBUG (ScheduledTrigger-206-thread-1) [    ] o

[...truncated too long message...]

tribDocExpirationUpdateProcessorTest, DistributedSpellCheckComponentTest, PreAnalyzedFieldTest, TestUseDocValuesAsStored, TestNoOpRegenerator, SolrMetricManagerTest, TestCoreDiscovery, TestNodeLostTrigger, UninvertDocValuesMergePolicyTest, ConcurrentCreateRoutedAliasTest, TestComplexPhraseLeadingWildcard, TestDelegationWithHadoopAuth, SearchHandlerTest, JvmMetricsTest, LIROnShardRestartTest, TestRandomDVFaceting, PeerSyncWithIndexFingerprintCachingTest, OverriddenZkACLAndCredentialsProvidersTest, StatsReloadRaceTest, ShowFileRequestHandlerTest, SearchHandlerTest, SuggesterTSTTest, DirectSolrSpellCheckerTest, CopyFieldTest, SmileWriterTest, FullHLLTest, TestFieldCacheSort, UtilsToolTest, DistanceFunctionTest, HdfsDirectoryFactoryTest, ClusterStateUpdateTest, SolrCmdDistributorTest, TestDistributedMissingSort, TestPrepRecovery, TestInPlaceUpdatesStandalone, DistributedFacetPivotSmallTest, LeaderElectionIntegrationTest, TestHighFrequencyDictionaryFactory, ClusterStateTest, TestMacroExpander, CoreSorterTest, TestLegacyField, CoreAdminHandlerTest, TestSolrFieldCacheBean, TestDirectoryFactory, HealthCheckHandlerTest, BlockJoinFacetRandomTest, TestJmxIntegration, BasicDistributedZk2Test, TestStressCloudBlindAtomicUpdates, TestComputePlanAction, CacheHeaderTest, ConvertedLegacyTest, TestJavabinTupleStreamParser, TestTriggerIntegration, TestBulkSchemaConcurrent, TestDynamicFieldCollectionResource, TestPolicyCloud, TestRawTransformer, ZkStateReaderTest, TestTrieFacet, ReturnFieldsTest, TestValueSourceCache, TestUpdate, ResponseLogComponentTest, TestSolrCloudWithDelegationTokens, MinimalSchemaTest, LargeFieldTest, DistributedQueryComponentOptimizationTest, WordBreakSolrSpellCheckerTest, MultiThreadedOCPTest, PingRequestHandlerTest, TestExactStatsCache, TestRealTimeGet, NumericFieldsTest, BigEndianAscendingWordSerializerTest, TestDocTermOrds, TestSolrConfigHandler, LeaderFailoverAfterPartitionTest, TestGenericDistributedQueue, LeaderElectionTest, TestBlendedInfixSuggestions, TestLazyCores, SpatialFilterTest, TestNonDefinedSimilarityFactory, ReplaceNodeNoTargetTest, TestRandomFlRTGCloud, TestDFRSimilarityFactory, TestFieldResource, TestSchemaVersionResource, UnloadDistributedZkTest, OpenCloseCoreStressTest, ShardRoutingTest, BasicZkTest, FullSolrCloudDistribCmdsTest, TestRandomFaceting, TestZkChroot, ShardRoutingCustomTest, TestFaceting, TestRecovery, TestHashPartitioner, TermVectorComponentDistributedTest, ZkControllerTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ZkControllerTest -Dtests.seed=7DFD2B6DF7CB6300 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=hu -Dtests.timezone=America/Rosario -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | ZkControllerTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [Overseer]
   [junit4]    > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.Overseer
   [junit4]    > 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]    > 	at org.apache.solr.cloud.Overseer.start(Overseer.java:545)
   [junit4]    > 	at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:850)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:393)
   [junit4]    > 	at org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:2063)
   [junit4]    > 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:331)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7DFD2B6DF7CB6300]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:301)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.ZkControllerTest: 
   [junit4]    >    1) Thread[id=15232, name=OverseerAutoScalingTriggerThread-72298237203054595-127.0.0.1:8983_solr-n_0000000001, state=RUNNABLE, group=Overseer autoscaling triggers]
   [junit4]    >         at java.lang.Object.notifyAll(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:652)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn.conLossPacket(ClientCnxn.java:674)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn.queuePacket(ClientCnxn.java:1452)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1405)
   [junit4]    >         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1213)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient$$Lambda$217/246968454.execute(Unknown Source)
   [junit4]    >         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]    >         at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7DFD2B6DF7CB6300]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=15232, name=OverseerAutoScalingTriggerThread-72298237203054595-127.0.0.1:8983_solr-n_0000000001, state=RUNNABLE, group=Overseer autoscaling triggers]
   [junit4]    >         at java.lang.Object.notifyAll(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:652)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn.conLossPacket(ClientCnxn.java:674)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn.queuePacket(ClientCnxn.java:1452)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1405)
   [junit4]    >         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1213)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:340)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient$$Lambda$217/246968454.execute(Unknown Source)
   [junit4]    >         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]    >         at org.apache.solr.client.solrj.impl.ZkDistribStateManager.getAutoScalingConfig(ZkDistribStateManager.java:176)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.DistribStateManager.getAutoScalingConfig(DistribStateManager.java:83)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:127)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7DFD2B6DF7CB6300]:0)
   [junit4] Completed [379/785 (4!)] on J2 in 105.86s, 4 tests, 1 failure, 2 errors, 1 skipped <<< FAILURES!

[...truncated 8809 lines...]
   [junit4] Suite: org.apache.solr.ltr.TestLTRReRankingPipeline
   [junit4]   2> 107311 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[2025C8687D18E4BE]) [    ] o.a.s.l.TestLTRReRankingPipeline rerank 1 documents 
   [junit4]   2> 107311 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[2025C8687D18E4BE]) [    ] o.a.s.l.TestLTRReRankingPipeline doc 0 in pos 0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTRReRankingPipeline -Dtests.method=testDifferentTopN -Dtests.seed=2025C8687D18E4BE -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=is -Dtests.timezone=Africa/Casablanca -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.11s J0 | TestLTRReRankingPipeline.testDifferentTopN <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1.0> but was:<0.0>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2025C8687D18E4BE:D184BA3848A32E2C]:0)
   [junit4]    > 	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=7, maxDocsPerChunk=68, blockSize=5), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=7, blockSize=5)), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@1ad349e3), locale=is, timezone=Africa/Casablanca
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=187581648,total=295698432
   [junit4]   2> NOTE: All tests run in this JVM: [TestLTROnSolrCloud, TestLTRWithSort, TestSelectiveWeightCreation, TestExternalValueFeatures, TestNoMatchSolrFeature, TestLTRReRankingPipeline]
   [junit4] Completed [27/37 (1!)] on J0 in 0.50s, 3 tests, 1 failure <<< FAILURES!

[...truncated 41964 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 13 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/13/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestLargeCluster

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster:     1) Thread[id=26466, name=AutoscalingActionExecutor-7414-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]         at java.util.stream.Sink$ChainedReference.end(Sink.java:258)         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:300)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:289)         at org.apache.solr.client.solrj.cloud.autoscaling.Row.addReplica(Row.java:122)         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:59)         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)         at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)         at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$645/355615214.run(Unknown Source)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$118/1804694261.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster: 
   1) Thread[id=26466, name=AutoscalingActionExecutor-7414-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]
        at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:300)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:289)
        at org.apache.solr.client.solrj.cloud.autoscaling.Row.addReplica(Row.java:122)
        at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:59)
        at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)
        at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)
        at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$645/355615214.run(Unknown Source)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$118/1804694261.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([7AB7A0FEDD558F]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestLargeCluster

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=26466, name=AutoscalingActionExecutor-7414-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]         at java.util.ArrayList.iterator(ArrayList.java:834)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:131)         at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:110)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)         at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:108)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:74)         at org.apache.solr.client.solrj.cloud.autoscaling.Row.copy(Row.java:91)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.lambda$getMatrixCopy$1(Policy.java:299)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session$$Lambda$564/489687804.apply(Unknown Source)         at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)         at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:300)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:289)         at org.apache.solr.client.solrj.cloud.autoscaling.Row.addReplica(Row.java:122)         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:59)         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)         at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)         at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$645/355615214.run(Unknown Source)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$118/1804694261.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=26466, name=AutoscalingActionExecutor-7414-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]
        at java.util.ArrayList.iterator(ArrayList.java:834)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:131)
        at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:110)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)
        at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:108)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:74)
        at org.apache.solr.client.solrj.cloud.autoscaling.Row.copy(Row.java:91)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.lambda$getMatrixCopy$1(Policy.java:299)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session$$Lambda$564/489687804.apply(Unknown Source)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:300)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:289)
        at org.apache.solr.client.solrj.cloud.autoscaling.Row.addReplica(Row.java:122)
        at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:59)
        at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)
        at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)
        at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$645/355615214.run(Unknown Source)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$118/1804694261.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([7AB7A0FEDD558F]:0)


FAILED:  org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([7AB7A0FEDD558F:2C48C02D7219230B]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:913)
	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

	request was:q=cat:h3l207k0ywkofvyx9547+66aul4w2hmjzhp4r69ji+g4sylcyjq6oe5e2gbbs7+lzhzj6oa4vax8awx9vbm+ftp4hwiqovgew66ehhuu&wt=xml
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:906)
	... 40 more


FAILED:  org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN

Error Message:
expected:<1.0> but was:<0.0>

Stack Trace:
java.lang.AssertionError: expected:<1.0> but was:<0.0>
	at __randomizedtesting.SeedInfo.seed([A9E2251DDA1B928D:5843574DEFA0581F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:443)
	at org.junit.Assert.assertEquals(Assert.java:512)
	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1831 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/test/temp/junit4-J0-20180316_224850_7676317647018401782754.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] codec: CheapBastard, pf: BlockTreeOrds, dvf: Lucene70
   [junit4] <<< JVM J0: EOF ----

[...truncated 11182 lines...]
   [junit4] Suite: org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_7AB7A0FEDD558F-001/init-core-data-001
   [junit4]   2> 1250289 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1250290 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1250290 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1250291 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 1250291 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1250324 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1250349 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 1250478 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1250531 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@73a6767b
   [junit4]   2> 1250543 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@73a6767b
   [junit4]   2> 1250543 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@73a6767b
   [junit4]   2> 1250545 INFO  (coreLoadExecutor-3373-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1250546 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1250581 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1250631 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1250824 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1250833 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 1250833 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@73a6767b
   [junit4]   2> 1250834 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1250834 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_7AB7A0FEDD558F-001/init-core-data-001/]
   [junit4]   2> 1250848 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=95.9033203125, floorSegmentMB=1.2001953125, forceMergeDeletesPctAllowed=15.09600521902838, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1250914 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1250914 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1250915 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1250915 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1250917 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=45, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.27456166311069163]
   [junit4]   2> 1250917 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5bb7c851[collection1] main]
   [junit4]   2> 1250918 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 1250937 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1251202 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1251203 INFO  (searcherExecutor-3374-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 1251203 INFO  (coreLoadExecutor-3373-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595139698549850112
   [junit4]   2> 1251203 INFO  (searcherExecutor-3374-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 1251203 INFO  (searcherExecutor-3374-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 1251203 INFO  (searcherExecutor-3374-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 1251204 INFO  (searcherExecutor-3374-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 1251204 INFO  (searcherExecutor-3374-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 1251204 INFO  (searcherExecutor-3374-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 1251204 INFO  (searcherExecutor-3374-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 1251204 INFO  (searcherExecutor-3374-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 1251206 INFO  (searcherExecutor-3374-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5bb7c851[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1251208 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 1251211 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testNoUniqueIdPassed-seed#[7AB7A0FEDD558F]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNoUniqueIdPassed
   [junit4]   2> 1251213 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testNoUniqueIdPassed-seed#[7AB7A0FEDD558F]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNoUniqueIdPassed
   [junit4]   2> 1251219 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testBasics
   [junit4]   2> 1251239 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1595139698587598848,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1251239 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@383c0d21 commitCommandVersion:1595139698587598848
   [junit4]   2> 1251251 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@4ece8df0[collection1] main]
   [junit4]   2> 1251252 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1251254 INFO  (searcherExecutor-3374-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4ece8df0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C1)))}
   [junit4]   2> 1251254 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 15
   [junit4]   2> 1251257 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1251262 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:human&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1251262 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Mr&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1251263 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:20&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1251263 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=name_s:Virat&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1251264 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=multiDefault:Delhi&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1251285 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1595139698635833344,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1251285 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@383c0d21 commitCommandVersion:1595139698635833344
   [junit4]   2> 1251288 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@159f983b[collection1] main]
   [junit4]   2> 1251288 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1251290 INFO  (searcherExecutor-3374-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@159f983b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(8.0.0):C1)))}
   [junit4]   2> 1251299 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 13
   [junit4]   2> 1251299 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1251300 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:human&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1251300 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:animal&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1251301 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Mr&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 1251301 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Dr&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1251302 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:20&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 1251302 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:40&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1251302 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=name_s:Virat&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 1251303 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=multiDefault:Delhi&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 1251303 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[7AB7A0FEDD558F]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testBasics
   [junit4]   2> 1251319 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[7AB7A0FEDD558F]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMultipleThreads
   [junit4]   2> 1251319 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[7AB7A0FEDD558F]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1251322 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[7AB7A0FEDD558F]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@7e2f093b[collection1] realtime]
   [junit4]   2> 1251322 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[7AB7A0FEDD558F]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 2
   [junit4]   2> 1251380 WARN  (Thread-2650) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1595139698722865153 actual=1595139698730205184Retrying with new version .... (1)
   [junit4]   2> 1251386 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[7AB7A0FEDD558F]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1595139698741739520,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1251386 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[7AB7A0FEDD558F]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@383c0d21 commitCommandVersion:1595139698741739520
   [junit4]   2> 1251389 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[7AB7A0FEDD558F]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@6752e4f1[collection1] main]
   [junit4]   2> 1251389 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[7AB7A0FEDD558F]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1251391 INFO  (searcherExecutor-3374-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6752e4f1[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_2(8.0.0):C10/9:delGen=1)))}
   [junit4]   2> 1251392 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[7AB7A0FEDD558F]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 5
   [junit4]   2> 1251392 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:10&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 1251393 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:h3l207k0ywkofvyx9547+66aul4w2hmjzhp4r69ji+g4sylcyjq6oe5e2gbbs7+lzhzj6oa4vax8awx9vbm+ftp4hwiqovgew66ehhuu&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 1251394 ERROR (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[7AB7A0FEDD558F]) [    ] o.a.s.SolrTestCaseJ4 REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]   2> 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2> 
   [junit4]   2> 	request was:q=cat:h3l207k0ywkofvyx9547+66aul4w2hmjzhp4r69ji+g4sylcyjq6oe5e2gbbs7+lzhzj6oa4vax8awx9vbm+ftp4hwiqovgew66ehhuu&wt=xml
   [junit4]   2> 1251398 ERROR (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[7AB7A0FEDD558F]) [    ] o.a.s.SolrTestCaseJ4 REQUEST FAILED: q=cat:h3l207k0ywkofvyx9547+66aul4w2hmjzhp4r69ji+g4sylcyjq6oe5e2gbbs7+lzhzj6oa4vax8awx9vbm+ftp4hwiqovgew66ehhuu&wt=xml:java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]   2> 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2> 
   [junit4]   2> 	request was:q=cat:h3l207k0ywkofvyx9547+66aul4w2hmjzhp4r69ji+g4sylcyjq6oe5e2gbbs7+lzhzj6oa4vax8awx9vbm+ftp4hwiqovgew66ehhuu&wt=xml
   [junit4]   2> 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:906)
   [junit4]   2> 	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:256)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 1251399 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[7AB7A0FEDD558F]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testMultipleThreads
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=AtomicUpdateProcessorFactoryTest -Dtests.method=testMultipleThreads -Dtests.seed=7AB7A0FEDD558F -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=no -Dtests.timezone=Pacific/Wake -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.10s J0 | AtomicUpdateProcessorFactoryTest.testMultipleThreads <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7AB7A0FEDD558F:2C48C02D7219230B]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:913)
   [junit4]    > 	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]    > 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]    > <response>
   [junit4]    > <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]    > </response>
   [junit4]    > 	request was:q=cat:h3l207k0ywkofvyx9547+66aul4w2hmjzhp4r69ji+g4sylcyjq6oe5e2gbbs7+lzhzj6oa4vax8awx9vbm+ftp4hwiqovgew66ehhuu&wt=xml
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:906)
   [junit4]    > 	... 40 more
   [junit4]   2> 1251419 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testWrongAtomicOpPassed-seed#[7AB7A0FEDD558F]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testWrongAtomicOpPassed
   [junit4]   2> 1251419 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testWrongAtomicOpPassed-seed#[7AB7A0FEDD558F]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testWrongAtomicOpPassed
   [junit4]   2> 1251419 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 1251420 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=335530116
   [junit4]   2> 1251420 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1251420 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@71a049ec: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@1b5403e1
   [junit4]   2> 1251424 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1251424 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2e57f426: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@2637ad13
   [junit4]   2> 1251430 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1251430 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5a425a27: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@531f67be
   [junit4]   2> 1251450 INFO  (coreCloseExecutor-3379-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7ab8ada4
   [junit4]   2> 1251451 INFO  (coreCloseExecutor-3379-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=2058923428
   [junit4]   2> 1251451 INFO  (coreCloseExecutor-3379-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@290a7f54: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@270b57c1
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_7AB7A0FEDD558F-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@7fca6d6a), locale=no, timezone=Pacific/Wake
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=99515248,total=519045120
   [junit4]   2> NOTE: All tests run in this JVM: [TestPullReplica, HdfsDirectoryTest, TestPhraseSuggestions, ReplaceNodeTest, DocExpirationUpdateProcessorFactoryTest, QueryResultKeyTest, TestRemoteStreaming, TestGroupingSearch, TestPerFieldSimilarityWithDefaultOverride, TestRandomFaceting, SynonymTokenizerTest, ComputePlanActionTest, HdfsCollectionsAPIDistributedZkTest, UUIDFieldTest, TestMacros, TestCSVLoader, TestComponentsName, TestApiFramework, RequestLoggingTest, TestManagedSynonymFilterFactory, CollectionsAPIDistributedZkTest, CollectionStateFormat2Test, LIRRollingUpdatesTest, DistributedFacetPivotLongTailTest, TestImpersonationWithHadoopAuth, TestStressInPlaceUpdates, TestDistribIDF, ZkSolrClientTest, DateRangeFieldTest, PluginInfoTest, BaseCdcrDistributedZkTest, SolrSlf4jReporterTest, TestComplexPhraseQParserPlugin, BadIndexSchemaTest, TestOverriddenPrefixQueryForCustomFieldType, TestSolrCloudWithKerberosAlt, TestConfigSetsAPIExclusivity, HdfsRestartWhileUpdatingTest, SimpleFacetsTest, ConfigSetsAPITest, TestQueryWrapperFilter, ZkFailoverTest, StressHdfsTest, TestMultiValuedNumericRangeQuery, MetricsConfigTest, XmlUpdateRequestHandlerTest, TestSSLRandomization, ExternalFileFieldSortTest, DeleteNodeTest, OutputWriterTest, RegexBytesRefFilterTest, CdcrReplicationDistributedZkTest, BlockCacheTest, TestJsonFacetRefinement, HdfsNNFailoverTest, SolrRequestParserTest, TestFieldCache, TestIBSimilarityFactory, TestSolrCLIRunExample, TestInitQParser, TestUninvertingReader, TestSerializedLuceneMatchVersion, TestSolrDeletionPolicy1, HLLSerializationTest, RollingRestartTest, TestFileDictionaryLookup, TestWriterPerf, RequestHandlersTest, AliasIntegrationTest, TestRestoreCore, SolrCoreCheckLockOnStartupTest, TestHashPartitioner, OpenExchangeRatesOrgProviderTest, MetricUtilsTest, TestCloudManagedSchema, BitVectorTest, TestTrackingShardHandlerFactory, PreAnalyzedUpdateProcessorTest, TestMinMaxOnMultiValuedField, TestEmbeddedSolrServerConstructors, SolrPluginUtilsTest, DebugComponentTest, PKIAuthenticationIntegrationTest, CdcrVersionReplicationTest, UpdateParamsTest, SolrMetricReporterTest, MergeStrategyTest, TestLeaderElectionZkExpiry, TestSchemaResource, TestHalfAndHalfDocValues, CurrencyRangeFacetCloudTest, TestDeleteCollectionOnDownNodes, HighlighterConfigTest, HttpTriggerListenerTest, AtomicUpdateProcessorFactoryTest]
   [junit4] Completed [288/785 (1!)] on J0 in 1.21s, 4 tests, 1 error <<< FAILURES!

[...truncated 1000 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestLargeCluster
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestLargeCluster_7AB7A0FEDD558F-001/init-core-data-001
   [junit4]   2> 3101614 INFO  (SUITE-TestLargeCluster-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 3101615 INFO  (SUITE-TestLargeCluster-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 3101616 INFO  (SUITE-TestLargeCluster-seed#[7AB7A0FEDD558F]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 3101617 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 3101617 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
   [junit4]   2> 3101618 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10033_solr, 127.0.0.1:10025_solr, 127.0.0.1:10026_solr, 127.0.0.1:10028_solr, 127.0.0.1:10036_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10037_solr, 127.0.0.1:10029_solr, 127.0.0.1:10035_solr, 127.0.0.1:10038_solr, 127.0.0.1:10030_solr, 127.0.0.1:10032_solr, 127.0.0.1:10027_solr]
   [junit4]   2> 3101618 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 3101618 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 3101619 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3101620 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
   [junit4]   2> 3101620 DEBUG (ScheduledTrigger-7407-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 37
   [junit4]   2> 3101645 DEBUG (ScheduledTrigger-7407-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101657 INFO  (TEST-TestLargeCluster.testNodeLost-seed#[7AB7A0FEDD558F]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLost
   [junit4]   2> 3101658 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 3101658 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 3101658 INFO  (TEST-TestLargeCluster.testNodeLost-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 3101661 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 3101664 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3101665 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
   [junit4]   2> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:165)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3101668 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[7AB7A0FEDD558F]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 3101669 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 3101669 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 3101670 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10118_solr, 127.0.0.1:10049_solr, 127.0.0.1:10104_solr, 127.0.0.1:10065_solr, 127.0.0.1:10120_solr, 127.0.0.1:10066_solr, 127.0.0.1:10121_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10096_solr, 127.0.0.1:10119_solr, 127.0.0.1:10033_solr, 127.0.0.1:10105_solr, 127.0.0.1:10047_solr, 127.0.0.1:10102_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 127.0.0.1:10123_solr, 127.0.0.1:10071_solr, 127.0.0.1:10076_solr, 127.0.0.1:10107_solr, 127.0.0.1:10055_solr, 127.0.0.1:10124_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10108_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10117_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10114_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10111_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10112_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10113_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10030_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10115_solr, 127.0.0.1:10069_solr, 127.0.0.1:10110_solr, 127.0.0.1:10062_solr, 127.0.0.1:10116_solr, 127.0.0.1:10122_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10106_solr, 127.0.0.1:10100_solr, 127.0.0.1:10037_solr, 127.0.0.1:10109_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10103_solr, 127.0.0.1:10072_solr, 127.0.0.1:10056_solr]
   [junit4]   2> 3101670 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 3101670 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 3101670 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3101670 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 3101670 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101695 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101716 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101738 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101758 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101775 DEBUG (simCloudManagerPool-7409-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3101776 DEBUG (simCloudManagerPool-7409-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 3101779 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101780 DEBUG (simCloudManagerPool-7409-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10118_solr, 127.0.0.1:10049_solr, 127.0.0.1:10104_solr, 127.0.0.1:10065_solr, 127.0.0.1:10120_solr, 127.0.0.1:10066_solr, 127.0.0.1:10121_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10096_solr, 127.0.0.1:10119_solr, 127.0.0.1:10033_solr, 127.0.0.1:10105_solr, 127.0.0.1:10047_solr, 127.0.0.1:10102_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 127.0.0.1:10123_solr, 127.0.0.1:10071_solr, 127.0.0.1:10076_solr, 127.0.0.1:10107_solr, 127.0.0.1:10055_solr, 127.0.0.1:10124_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10108_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10117_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10114_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10111_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10112_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10113_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10030_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10115_solr, 127.0.0.1:10069_solr, 127.0.0.1:10110_solr, 127.0.0.1:10062_solr, 127.0.0.1:10116_solr, 127.0.0.1:10122_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10106_solr, 127.0.0.1:10100_solr, 127.0.0.1:10037_solr, 127.0.0.1:10109_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10103_solr, 127.0.0.1:10072_solr, 127.0.0.1:10056_solr]
   [junit4]   2> 3101780 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 3101780 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3101780 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 3101784 DEBUG (simCloudManagerPool-7409-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3101784 DEBUG (simCloudManagerPool-7409-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 3101784 DEBUG (simCloudManagerPool-7409-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10118_solr, 127.0.0.1:10049_solr, 127.0.0.1:10104_solr, 127.0.0.1:10065_solr, 127.0.0.1:10120_solr, 127.0.0.1:10066_solr, 127.0.0.1:10121_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10096_solr, 127.0.0.1:10119_solr, 127.0.0.1:10033_solr, 127.0.0.1:10105_solr, 127.0.0.1:10047_solr, 127.0.0.1:10102_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 127.0.0.1:10123_solr, 127.0.0.1:10071_solr, 127.0.0.1:10076_solr, 127.0.0.1:10107_solr, 127.0.0.1:10055_solr, 127.0.0.1:10124_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10108_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10117_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10114_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10111_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10112_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10113_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10030_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10115_solr, 127.0.0.1:10069_solr, 127.0.0.1:10110_solr, 127.0.0.1:10062_solr, 127.0.0.1:10116_solr, 127.0.0.1:10122_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10106_solr, 127.0.0.1:10100_solr, 127.0.0.1:10037_solr, 127.0.0.1:10109_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10103_solr, 127.0.0.1:10072_solr, 127.0.0.1:10056_solr]
   [junit4]   2> 3101785 DEBUG (simCloudManagerPool-7409-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10118_solr, 127.0.0.1:10049_solr, 127.0.0.1:10104_solr, 127.0.0.1:10065_solr, 127.0.0.1:10120_solr, 127.0.0.1:10066_solr, 127.0.0.1:10121_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10096_solr, 127.0.0.1:10119_solr, 127.0.0.1:10033_solr, 127.0.0.1:10105_solr, 127.0.0.1:10047_solr, 127.0.0.1:10102_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 127.0.0.1:10123_solr, 127.0.0.1:10071_solr, 127.0.0.1:10076_solr, 127.0.0.1:10107_solr, 127.0.0.1:10055_solr, 127.0.0.1:10124_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10108_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10117_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10114_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10111_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10112_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10113_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10030_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10115_solr, 127.0.0.1:10069_solr, 127.0.0.1:10110_solr, 127.0.0.1:10062_solr, 127.0.0.1:10116_solr, 127.0.0.1:10122_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10106_solr, 127.0.0.1:10100_solr, 127.0.0.1:10037_solr, 127.0.0.1:10109_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10103_solr, 127.0.0.1:10072_solr, 127.0.0.1:10056_solr]
   [junit4]   2> 3101785 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 3101785 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3101785 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 3101785 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3101799 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101809 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3101820 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101829 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3101840 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101849 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3101861 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101870 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3101881 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101890 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3101901 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101910 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3101921 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101931 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3101942 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101951 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3101962 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101971 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3101982 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3101992 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102003 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102012 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102023 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102032 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102044 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102053 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102064 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102073 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102084 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102093 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102105 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102114 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102125 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102134 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102145 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102154 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102166 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102175 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102186 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102195 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102206 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102215 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102227 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102235 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102247 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102256 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102267 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102276 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102288 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102297 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102308 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102317 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102329 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102337 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102349 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102357 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102369 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102378 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102390 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102398 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102410 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102418 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102430 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102439 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102451 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102459 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102471 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102479 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102491 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102500 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102512 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102520 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102532 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102540 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102553 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102561 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102573 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102581 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102593 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102601 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102614 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102622 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102634 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102642 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102654 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102663 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102675 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102683 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102695 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102703 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102715 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102724 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102735 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102744 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102756 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102764 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102776 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102785 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102796 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102805 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102817 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102825 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102837 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102846 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102857 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102866 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102878 DEBUG (ScheduledTrigger-7410-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102886 DEBUG (ScheduledTrigger-7410-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102898 DEBUG (ScheduledTrigger-7410-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3102907 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 100
   [junit4]   2> 3102918 DEBUG (ScheduledTrigger-7410-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 31029

[...truncated too long message...]

NodeMutatorTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverseerModifyCollectionTest, OverseerRolesTest, OverseerStatusTest, OverseerTaskQueueTest, RestartWhileUpdatingTest, SolrCloudExampleTest, TestCloudDeleteByQuery, TestCloudInspectUtil, TestCloudPivotFacet, TestCloudPseudoReturnFields, TestCloudSearcherWarming, TestLeaderElectionWithEmptyReplica, TestRebalanceLeaders, TestTolerantUpdateProcessorCloud, TestTolerantUpdateProcessorRandomCloud, TestUtilizeNode, TlogReplayBufferedWhileIndexingTest, TriLevelCompositeIdRoutingTest, VMParamsZkACLAndCredentialsProvidersTest, ZkShardTermsTest, AssignTest, ConcurrentDeleteAndCreateCollectionTest, CustomCollectionTest, TestLocalFSCloudBackupRestore, TestRequestStatusCollectionAPI, AutoAddReplicasPlanActionTest, AutoScalingHandlerTest, ExecutePlanActionTest, SearchRateTriggerTest, TestClusterStateProvider, TestComputePlanAction, TestDistribStateManager, TestExecutePlanAction, TestLargeCluster]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLargeCluster -Dtests.seed=7AB7A0FEDD558F -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ja -Dtests.timezone=Asia/Aqtau -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J2 | TestLargeCluster (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster: 
   [junit4]    >    1) Thread[id=26466, name=AutoscalingActionExecutor-7414-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]
   [junit4]    >         at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
   [junit4]    >         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
   [junit4]    >         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
   [junit4]    >         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
   [junit4]    >         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
   [junit4]    >         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:300)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:289)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Row.addReplica(Row.java:122)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:59)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$645/355615214.run(Unknown Source)
   [junit4]    >         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]    >         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$118/1804694261.run(Unknown Source)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7AB7A0FEDD558F]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=26466, name=AutoscalingActionExecutor-7414-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]
   [junit4]    >         at java.util.ArrayList.iterator(ArrayList.java:834)
   [junit4]    >         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:131)
   [junit4]    >         at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:110)
   [junit4]    >         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)
   [junit4]    >         at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:108)
   [junit4]    >         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)
   [junit4]    >         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:74)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Row.copy(Row.java:91)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.lambda$getMatrixCopy$1(Policy.java:299)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session$$Lambda$564/489687804.apply(Unknown Source)
   [junit4]    >         at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
   [junit4]    >         at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
   [junit4]    >         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
   [junit4]    >         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
   [junit4]    >         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
   [junit4]    >         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
   [junit4]    >         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:300)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:289)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Row.addReplica(Row.java:122)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:59)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$645/355615214.run(Unknown Source)
   [junit4]    >         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]    >         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$118/1804694261.run(Unknown Source)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7AB7A0FEDD558F]:0)
   [junit4] Completed [597/785 (2!)] on J2 in 63.67s, 4 tests, 2 errors <<< FAILURES!

[...truncated 8043 lines...]
   [junit4] Suite: org.apache.solr.ltr.TestLTRReRankingPipeline
   [junit4]   2> 21265 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[A9E2251DDA1B928D]) [    ] o.a.s.l.TestLTRReRankingPipeline rerank 1 documents 
   [junit4]   2> 21266 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[A9E2251DDA1B928D]) [    ] o.a.s.l.TestLTRReRankingPipeline doc 0 in pos 0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTRReRankingPipeline -Dtests.method=testDifferentTopN -Dtests.seed=A9E2251DDA1B928D -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=el-GR -Dtests.timezone=America/Grenada -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.06s J0 | TestLTRReRankingPipeline.testDifferentTopN <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1.0> but was:<0.0>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A9E2251DDA1B928D:5843574DEFA0581F]:0)
   [junit4]    > 	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {field=PostingsFormat(name=Asserting), id=Lucene50(blocksize=128), final-score=Lucene50(blocksize=128)}, docValues:{final-score=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1344, maxMBSortInHeap=5.749721465122952, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@6767334c), locale=el-GR, timezone=America/Grenada
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=162991200,total=289406976
   [junit4]   2> NOTE: All tests run in this JVM: [TestUserTermScorereQDF, TestModelManagerPersistence, TestExternalValueFeatures, TestLTRReRankingPipeline]
   [junit4] Completed [16/37 (1!)] on J0 in 0.24s, 3 tests, 1 failure <<< FAILURES!

[...truncated 41997 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 12 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/12/

6 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue

Error Message:
action wasn't interrupted

Stack Trace:
java.lang.AssertionError: action wasn't interrupted
	at __randomizedtesting.SeedInfo.seed([7DD583F7A5B89E45:B460C159ACDF58B0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue(TriggerIntegrationTest.java:752)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:44482_solr, 127.0.0.1:60434_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/8)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node2":{           "core":"testSimple1_shard1_replica_n1",           "base_url":"https://127.0.0.1:60434/solr",           "node_name":"127.0.0.1:60434_solr",           "state":"active",           "type":"NRT",           "leader":"true"},         "core_node5":{           "core":"testSimple1_shard1_replica_n3",           "base_url":"https://127.0.0.1:53021/solr",           "node_name":"127.0.0.1:53021_solr",           "state":"down",           "type":"NRT"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node7":{           "core":"testSimple1_shard2_replica_n4",           "base_url":"https://127.0.0.1:60434/solr",           "node_name":"127.0.0.1:60434_solr",           "state":"active",           "type":"NRT",           "leader":"true"},         "core_node8":{           "core":"testSimple1_shard2_replica_n6",           "base_url":"https://127.0.0.1:53021/solr",           "node_name":"127.0.0.1:53021_solr",           "state":"down",           "type":"NRT"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:44482_solr, 127.0.0.1:60434_solr]
Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/8)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"testSimple1_shard1_replica_n1",
          "base_url":"https://127.0.0.1:60434/solr",
          "node_name":"127.0.0.1:60434_solr",
          "state":"active",
          "type":"NRT",
          "leader":"true"},
        "core_node5":{
          "core":"testSimple1_shard1_replica_n3",
          "base_url":"https://127.0.0.1:53021/solr",
          "node_name":"127.0.0.1:53021_solr",
          "state":"down",
          "type":"NRT"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "core":"testSimple1_shard2_replica_n4",
          "base_url":"https://127.0.0.1:60434/solr",
          "node_name":"127.0.0.1:60434_solr",
          "state":"active",
          "type":"NRT",
          "leader":"true"},
        "core_node8":{
          "core":"testSimple1_shard2_replica_n6",
          "base_url":"https://127.0.0.1:53021/solr",
          "node_name":"127.0.0.1:53021_solr",
          "state":"down",
          "type":"NRT"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
	at __randomizedtesting.SeedInfo.seed([7DD583F7A5B89E45:4566A709824B4A94]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:269)
	at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:94)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:35812_solr, 127.0.0.1:51729_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/10)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "dataDir":"hdfs://localhost:40572/data/testSimple1/core_node3/data/",           "base_url":"https://127.0.0.1:51729/solr",           "node_name":"127.0.0.1:51729_solr",           "type":"NRT",           "ulogDir":"hdfs://localhost:40572/data/testSimple1/core_node3/data/tlog",           "core":"testSimple1_shard1_replica_n1",           "shared_storage":"true",           "state":"active",           "leader":"true"},         "core_node5":{           "dataDir":"hdfs://localhost:40572/data/testSimple1/core_node5/data/",           "base_url":"https://127.0.0.1:45035/solr",           "node_name":"127.0.0.1:45035_solr",           "type":"NRT",           "ulogDir":"hdfs://localhost:40572/data/testSimple1/core_node5/data/tlog",           "core":"testSimple1_shard1_replica_n2",           "shared_storage":"true",           "state":"down"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node7":{           "dataDir":"hdfs://localhost:40572/data/testSimple1/core_node7/data/",           "base_url":"https://127.0.0.1:51729/solr",           "node_name":"127.0.0.1:51729_solr",           "type":"NRT",           "ulogDir":"hdfs://localhost:40572/data/testSimple1/core_node7/data/tlog",           "core":"testSimple1_shard2_replica_n4",           "shared_storage":"true",           "state":"active",           "leader":"true"},         "core_node8":{           "dataDir":"hdfs://localhost:40572/data/testSimple1/core_node8/data/",           "base_url":"https://127.0.0.1:45035/solr",           "node_name":"127.0.0.1:45035_solr",           "type":"NRT",           "ulogDir":"hdfs://localhost:40572/data/testSimple1/core_node8/data/tlog",           "core":"testSimple1_shard2_replica_n6",           "shared_storage":"true",           "state":"down"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:35812_solr, 127.0.0.1:51729_solr]
Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/10)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://localhost:40572/data/testSimple1/core_node3/data/",
          "base_url":"https://127.0.0.1:51729/solr",
          "node_name":"127.0.0.1:51729_solr",
          "type":"NRT",
          "ulogDir":"hdfs://localhost:40572/data/testSimple1/core_node3/data/tlog",
          "core":"testSimple1_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node5":{
          "dataDir":"hdfs://localhost:40572/data/testSimple1/core_node5/data/",
          "base_url":"https://127.0.0.1:45035/solr",
          "node_name":"127.0.0.1:45035_solr",
          "type":"NRT",
          "ulogDir":"hdfs://localhost:40572/data/testSimple1/core_node5/data/tlog",
          "core":"testSimple1_shard1_replica_n2",
          "shared_storage":"true",
          "state":"down"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://localhost:40572/data/testSimple1/core_node7/data/",
          "base_url":"https://127.0.0.1:51729/solr",
          "node_name":"127.0.0.1:51729_solr",
          "type":"NRT",
          "ulogDir":"hdfs://localhost:40572/data/testSimple1/core_node7/data/tlog",
          "core":"testSimple1_shard2_replica_n4",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node8":{
          "dataDir":"hdfs://localhost:40572/data/testSimple1/core_node8/data/",
          "base_url":"https://127.0.0.1:45035/solr",
          "node_name":"127.0.0.1:45035_solr",
          "type":"NRT",
          "ulogDir":"hdfs://localhost:40572/data/testSimple1/core_node8/data/tlog",
          "core":"testSimple1_shard2_replica_n6",
          "shared_storage":"true",
          "state":"down"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
	at __randomizedtesting.SeedInfo.seed([7DD583F7A5B89E45:4566A709824B4A94]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:269)
	at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:94)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([7DD583F7A5B89E45:51E7F47A297CE8C1]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:908)
	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:260)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

	request was:q=int_i:22&wt=xml
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:901)
	... 40 more


FAILED:  org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN

Error Message:
expected:<1.0> but was:<0.0>

Stack Trace:
java.lang.AssertionError: expected:<1.0> but was:<0.0>
	at __randomizedtesting.SeedInfo.seed([5394B1BAA58B6095:A235C3EA9030AA07]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:443)
	at org.junit.Assert.assertEquals(Assert.java:512)
	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart

Error Message:
IOException occured when talking to server at: http://127.0.0.1:53670/solr/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:53670/solr/collection1
	at __randomizedtesting.SeedInfo.seed([7DD583F7A5B89E45:A52247130E635C19]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
	at org.apache.solr.handler.TestReplicationHandler.index(TestReplicationHandler.java:180)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart(TestReplicationHandler.java:643)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:53670 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 47 more




Build Log:
[...truncated 12450 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest
   [junit4]   2> 309830 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7DD583F7A5B89E45-001/init-core-data-001
   [junit4]   2> 309831 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 309831 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 309833 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 309834 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7DD583F7A5B89E45-001/tempDir-001
   [junit4]   2> 309834 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 309834 INFO  (Thread-863) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 309834 INFO  (Thread-863) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 309841 ERROR (Thread-863) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 309934 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:40895
   [junit4]   2> 309938 INFO  (zkConnectionManagerCallback-656-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 309944 INFO  (jetty-launcher-653-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 309949 INFO  (jetty-launcher-653-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 309950 INFO  (jetty-launcher-653-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 309950 INFO  (jetty-launcher-653-thread-1) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 309950 INFO  (jetty-launcher-653-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@b68215a{/solr,null,AVAILABLE}
   [junit4]   2> 309951 INFO  (jetty-launcher-653-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@600daca3{SSL,[ssl, http/1.1]}{127.0.0.1:35812}
   [junit4]   2> 309952 INFO  (jetty-launcher-653-thread-1) [    ] o.e.j.s.Server Started @312486ms
   [junit4]   2> 309952 INFO  (jetty-launcher-653-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35812}
   [junit4]   2> 309952 ERROR (jetty-launcher-653-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 309952 INFO  (jetty-launcher-653-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 309952 INFO  (jetty-launcher-653-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 309952 INFO  (jetty-launcher-653-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 309952 INFO  (jetty-launcher-653-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-12T19:50:51.514Z
   [junit4]   2> 309953 INFO  (jetty-launcher-653-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 309955 INFO  (jetty-launcher-653-thread-3) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 309961 INFO  (jetty-launcher-653-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 309961 INFO  (jetty-launcher-653-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 309961 INFO  (jetty-launcher-653-thread-3) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 309962 INFO  (jetty-launcher-653-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@36dcf9c9{/solr,null,AVAILABLE}
   [junit4]   2> 309962 INFO  (jetty-launcher-653-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@aac6816{SSL,[ssl, http/1.1]}{127.0.0.1:51729}
   [junit4]   2> 309962 INFO  (jetty-launcher-653-thread-3) [    ] o.e.j.s.Server Started @312497ms
   [junit4]   2> 309962 INFO  (jetty-launcher-653-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=51729}
   [junit4]   2> 309963 ERROR (jetty-launcher-653-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 309963 INFO  (jetty-launcher-653-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 309963 INFO  (jetty-launcher-653-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 309963 INFO  (jetty-launcher-653-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 309963 INFO  (jetty-launcher-653-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-12T19:50:51.525Z
   [junit4]   2> 309968 INFO  (zkConnectionManagerCallback-658-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 309968 INFO  (jetty-launcher-653-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 309976 INFO  (jetty-launcher-653-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 309976 INFO  (jetty-launcher-653-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 309976 INFO  (jetty-launcher-653-thread-2) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 309976 INFO  (zkConnectionManagerCallback-660-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 309977 INFO  (jetty-launcher-653-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3c752606{/solr,null,AVAILABLE}
   [junit4]   2> 309978 INFO  (jetty-launcher-653-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@eb23a85{SSL,[ssl, http/1.1]}{127.0.0.1:45035}
   [junit4]   2> 309978 INFO  (jetty-launcher-653-thread-2) [    ] o.e.j.s.Server Started @312513ms
   [junit4]   2> 309978 INFO  (jetty-launcher-653-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45035}
   [junit4]   2> 309978 ERROR (jetty-launcher-653-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 309978 INFO  (jetty-launcher-653-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 309978 INFO  (jetty-launcher-653-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 309978 INFO  (jetty-launcher-653-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 309978 INFO  (jetty-launcher-653-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-12T19:50:51.540Z
   [junit4]   2> 309979 INFO  (jetty-launcher-653-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 309983 INFO  (jetty-launcher-653-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40895/solr
   [junit4]   2> 309989 INFO  (jetty-launcher-653-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40895/solr
   [junit4]   2> 309992 INFO  (zkConnectionManagerCallback-662-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 309993 INFO  (zkConnectionManagerCallback-666-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 309993 INFO  (jetty-launcher-653-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 309995 INFO  (zkConnectionManagerCallback-670-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 310001 INFO  (zkConnectionManagerCallback-672-thread-1-processing-n:127.0.0.1:35812_solr) [n:127.0.0.1:35812_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 310025 INFO  (zkConnectionManagerCallback-674-thread-1-processing-n:127.0.0.1:51729_solr) [n:127.0.0.1:51729_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 310026 INFO  (jetty-launcher-653-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40895/solr
   [junit4]   2> 310040 INFO  (zkConnectionManagerCallback-679-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 310047 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x104ebd01d8b0008, likely client has closed socket
   [junit4]   2> 310081 INFO  (zkConnectionManagerCallback-682-thread-1-processing-n:127.0.0.1:45035_solr) [n:127.0.0.1:45035_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 310117 INFO  (jetty-launcher-653-thread-2) [n:127.0.0.1:45035_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 310118 INFO  (jetty-launcher-653-thread-2) [n:127.0.0.1:45035_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:45035_solr
   [junit4]   2> 310120 INFO  (jetty-launcher-653-thread-2) [n:127.0.0.1:45035_solr    ] o.a.s.c.Overseer Overseer (id=73442773026144265-127.0.0.1:45035_solr-n_0000000000) starting
   [junit4]   2> 310174 INFO  (zkConnectionManagerCallback-687-thread-1-processing-n:127.0.0.1:45035_solr) [n:127.0.0.1:45035_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 310177 INFO  (jetty-launcher-653-thread-2) [n:127.0.0.1:45035_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40895/solr ready
   [junit4]   2> 310193 INFO  (jetty-launcher-653-thread-2) [n:127.0.0.1:45035_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45035_solr
   [junit4]   2> 310193 INFO  (jetty-launcher-653-thread-3) [n:127.0.0.1:51729_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 310194 DEBUG (OverseerAutoScalingTriggerThread-73442773026144265-127.0.0.1:45035_solr-n_0000000000) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 310196 DEBUG (OverseerAutoScalingTriggerThread-73442773026144265-127.0.0.1:45035_solr-n_0000000000) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 310218 INFO  (jetty-launcher-653-thread-3) [n:127.0.0.1:51729_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51729_solr
   [junit4]   2> 310218 DEBUG (OverseerAutoScalingTriggerThread-73442773026144265-127.0.0.1:45035_solr-n_0000000000) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: []
   [junit4]   2> 310218 DEBUG (OverseerAutoScalingTriggerThread-73442773026144265-127.0.0.1:45035_solr-n_0000000000) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 310218 DEBUG (OverseerAutoScalingTriggerThread-73442773026144265-127.0.0.1:45035_solr-n_0000000000) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 310223 INFO  (jetty-launcher-653-thread-1) [n:127.0.0.1:35812_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 310225 INFO  (jetty-launcher-653-thread-1) [n:127.0.0.1:35812_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 310228 INFO  (jetty-launcher-653-thread-1) [n:127.0.0.1:35812_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35812_solr
   [junit4]   2> 310237 INFO  (zkCallback-681-thread-1-processing-n:127.0.0.1:45035_solr) [n:127.0.0.1:45035_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 310273 INFO  (zkCallback-673-thread-1-processing-n:127.0.0.1:51729_solr) [n:127.0.0.1:51729_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 310273 INFO  (zkCallback-686-thread-1-processing-n:127.0.0.1:45035_solr) [n:127.0.0.1:45035_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 310274 INFO  (zkCallback-673-thread-2-processing-n:127.0.0.1:51729_solr) [n:127.0.0.1:51729_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 310285 DEBUG (ScheduledTrigger-1412-thread-1) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 310298 DEBUG (OverseerAutoScalingTriggerThread-73442773026144265-127.0.0.1:45035_solr-n_0000000000) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 310298 DEBUG (OverseerAutoScalingTriggerThread-73442773026144265-127.0.0.1:45035_solr-n_0000000000) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 310306 INFO  (zkCallback-671-thread-1-processing-n:127.0.0.1:35812_solr) [n:127.0.0.1:35812_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 310352 INFO  (jetty-launcher-653-thread-3) [n:127.0.0.1:51729_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51729.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@383c60a
   [junit4]   2> 310358 INFO  (jetty-launcher-653-thread-2) [n:127.0.0.1:45035_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45035.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@383c60a
   [junit4]   2> 310365 INFO  (jetty-launcher-653-thread-3) [n:127.0.0.1:51729_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51729.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@383c60a
   [junit4]   2> 310365 INFO  (jetty-launcher-653-thread-3) [n:127.0.0.1:51729_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51729.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@383c60a
   [junit4]   2> 310367 INFO  (jetty-launcher-653-thread-3) [n:127.0.0.1:51729_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7DD583F7A5B89E45-001/tempDir-001/node3/.
   [junit4]   2> 310370 INFO  (jetty-launcher-653-thread-2) [n:127.0.0.1:45035_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45035.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@383c60a
   [junit4]   2> 310370 INFO  (jetty-launcher-653-thread-2) [n:127.0.0.1:45035_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45035.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@383c60a
   [junit4]   2> 310371 INFO  (jetty-launcher-653-thread-2) [n:127.0.0.1:45035_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7DD583F7A5B89E45-001/tempDir-001/node2/.
   [junit4]   2> 310398 INFO  (jetty-launcher-653-thread-1) [n:127.0.0.1:35812_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35812.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@383c60a
   [junit4]   2> 310405 INFO  (zkConnectionManagerCallback-691-thread-1-processing-n:127.0.0.1:51729_solr) [n:127.0.0.1:51729_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 310406 INFO  (jetty-launcher-653-thread-3) [n:127.0.0.1:51729_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 310407 INFO  (jetty-launcher-653-thread-3) [n:127.0.0.1:51729_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40895/solr ready
   [junit4]   2> 310416 INFO  (jetty-launcher-653-thread-1) [n:127.0.0.1:35812_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35812.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@383c60a
   [junit4]   2> 310416 INFO  (jetty-launcher-653-thread-1) [n:127.0.0.1:35812_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35812.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@383c60a
   [junit4]   2> 310418 INFO  (jetty-launcher-653-thread-1) [n:127.0.0.1:35812_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7DD583F7A5B89E45-001/tempDir-001/node1/.
   [junit4]   2> 310447 INFO  (zkConnectionManagerCallback-695-thread-1-processing-n:127.0.0.1:35812_solr) [n:127.0.0.1:35812_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 310448 INFO  (jetty-launcher-653-thread-1) [n:127.0.0.1:35812_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 310449 INFO  (jetty-launcher-653-thread-1) [n:127.0.0.1:35812_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40895/solr ready
   [junit4]   2> 310562 INFO  (zkConnectionManagerCallback-697-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 310581 INFO  (zkConnectionManagerCallback-701-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 310582 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 310583 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40895/solr ready
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 310652 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 310677 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 310684 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 310701 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_52497_hdfs____.t9s2ip/webapp
   [junit4]   2> 311182 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:52497
   [junit4]   2> 311293 DEBUG (ScheduledTrigger-1412-thread-1) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 311353 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 311354 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 311383 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_36502_datanode____.fz5ahx/webapp
   [junit4]   2> 311977 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36502
   [junit4]   2> 312029 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 312030 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 312052 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_39251_datanode____.azroc1/webapp
   [junit4]   2> 312100 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7DD583F7A5B89E45-001/tempDir-002/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7DD583F7A5B89E45-001/tempDir-002/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:40572) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 312114 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x4b1738de86a895: from storage DS-77938411-5f42-4694-89d7-7f77f44ee498 node DatanodeRegistration(127.0.0.1:51757, datanodeUuid=d2e4280e-4d48-4c7a-a4ba-129dcc7e2ecb, infoPort=57793, infoSecurePort=0, ipcPort=52782, storageInfo=lv=-56;cid=testClusterID;nsid=1784579511;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 312114 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x4b1738de86a895: from storage DS-cbc87cc0-a176-474c-9082-fe115cd6bb89 node DatanodeRegistration(127.0.0.1:51757, datanodeUuid=d2e4280e-4d48-4c7a-a4ba-129dcc7e2ecb, infoPort=57793, infoSecurePort=0, ipcPort=52782, storageInfo=lv=-56;cid=testClusterID;nsid=1784579511;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 313186 DEBUG (ScheduledTrigger-1412-thread-2) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 313453 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39251
   [junit4]   2> 313661 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7DD583F7A5B89E45-001/tempDir-002/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7DD583F7A5B89E45-001/tempDir-002/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:40572) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 313686 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x4b17393c4a614d: from storage DS-a925587e-2a4d-4721-8240-e45db73c015a node DatanodeRegistration(127.0.0.1:47863, datanodeUuid=114b061a-dfa5-46f0-a59f-9af88439c0d3, infoPort=52075, infoSecurePort=0, ipcPort=34782, storageInfo=lv=-56;cid=testClusterID;nsid=1784579511;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 313687 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x4b17393c4a614d: from storage DS-47ed794c-64fa-46f0-806c-d6f6fd458d50 node DatanodeRegistration(127.0.0.1:47863, datanodeUuid=114b061a-dfa5-46f0-a59f-9af88439c0d3, infoPort=52075, infoSecurePort=0, ipcPort=34782, storageInfo=lv=-56;cid=testClusterID;nsid=1784579511;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 313813 INFO  (TEST-HdfsAutoAddReplicasIntegrationTest.testSimple-seed#[7DD583F7A5B89E45]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSimple
   [junit4]   2> 313851 INFO  (qtp668097388-3253) [n:127.0.0.1:45035_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&maxShardsPerNode=2&autoAddReplicas=true&name=testSimple1&nrtReplicas=2&action=CREATE&numShards=2&createNodeSet=127.0.0.1:45035_solr,127.0.0.1:51729_solr&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 313854 INFO  (OverseerThreadFactory-1414-thread-1-processing-n:127.0.0.1:45035_solr) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testSimple1
   [junit4]   2> 313964 INFO  (OverseerStateUpdate-73442773026144265-127.0.0.1:45035_solr-n_0000000000) [n:127.0.0.1:45035_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testSimple1_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:51729/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 313969 INFO  (OverseerStateUpdate-73442773026144265-127.0.0.1:45035_solr-n_0000000000) [n:127.0.0.1:45035_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testSimple1_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:45035/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 313972 INFO  (OverseerStateUpdate-73442773026144265-127.0.0.1:45035_solr-n_0000000000) [n:127.0.0.1:45035_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testSimple1_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:51729/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 313976 INFO  (OverseerStateUpdate-73442773026144265-127.0.0.1:45035_solr-n_0000000000) [n:127.0.0.1:45035_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testSimple1_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:45035/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 314187 DEBUG (ScheduledTrigger-1412-thread-4) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 314193 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n2&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 314197 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 314199 INFO  (qtp668097388-3254) [n:127.0.0.1:45035_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n6&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 314243 INFO  (qtp1950758694-3240) [n:127.0.0.1:51729_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n1&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 314243 INFO  (qtp1950758694-3240) [n:127.0.0.1:51729_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 314244 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n4&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 314353 INFO  (zkCallback-673-thread-2-processing-n:127.0.0.1:51729_solr) [n:127.0.0.1:51729_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 314353 INFO  (zkCallback-681-thread-2-processing-n:127.0.0.1:45035_solr) [n:127.0.0.1:45035_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 314353 INFO  (zkCallback-673-thread-1-processing-n:127.0.0.1:51729_solr) [n:127.0.0.1:51729_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 314353 INFO  (zkCallback-681-thread-1-processing-n:127.0.0.1:45035_solr) [n:127.0.0.1:45035_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 314354 INFO  (zkCallback-673-thread-3-processing-n:127.0.0.1:51729_solr) [n:127.0.0.1:51729_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 315187 DEBUG (ScheduledTrigger-1412-thread-4) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 315226 INFO  (qtp668097388-3254) [n:127.0.0.1:45035_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 315238 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 315238 INFO  (qtp668097388-3254) [n:127.0.0.1:45035_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.IndexSchema [testSimple1_shard2_replica_n6] Schema name=minimal
   [junit4]   2> 315242 INFO  (qtp668097388-3254) [n:127.0.0.1:45035_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 315242 INFO  (qtp668097388-3254) [n:127.0.0.1:45035_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard2_replica_n6' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 315243 INFO  (qtp668097388-3254) [n:127.0.0.1:45035_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45035.solr.core.testSimple1.shard2.replica_n6' (registry 'solr.core.testSimple1.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@383c60a
   [junit4]   2> 315249 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.IndexSchema [testSimple1_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 315251 INFO  (qtp668097388-3254) [n:127.0.0.1:45035_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:40572/data
   [junit4]   2> 315251 INFO  (qtp668097388-3254) [n:127.0.0.1:45035_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 315251 INFO  (qtp668097388-3254) [n:127.0.0.1:45035_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 315251 INFO  (qtp668097388-3254) [n:127.0.0.1:45035_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrCore [[testSimple1_shard2_replica_n6] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7DD583F7A5B89E45-001/tempDir-001/node2/testSimple1_shard2_replica_n6], dataDir=[hdfs://localhost:40572/data/testSimple1/core_node8/data/]
   [junit4]   2> 315252 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 315252 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard1_replica_n2' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 315253 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45035.solr.core.testSimple1.shard1.replica_n2' (registry 'solr.core.testSimple1.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@383c60a
   [junit4]   2> 315253 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:40572/data
   [junit4]   2> 315253 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 315253 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 315253 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrCore [[testSimple1_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7DD583F7A5B89E45-001/tempDir-001/node2/testSimple1_shard1_replica_n2], dataDir=[hdfs://localhost:40572/data/testSimple1/core_node5/data/]
   [junit4]   2> 315256 INFO  (qtp668097388-3254) [n:127.0.0.1:45035_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40572/data/testSimple1/core_node8/data/snapshot_metadata
   [junit4]   2> 315256 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40572/data/testSimple1/core_node5/data/snapshot_metadata
   [junit4]   2> 315261 INFO  (qtp1950758694-3240) [n:127.0.0.1:51729_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 315261 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 315276 INFO  (qtp1950758694-3240) [n:127.0.0.1:51729_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.IndexSchema [testSimple1_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 315281 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.IndexSchema [testSimple1_shard2_replica_n4] Schema name=minimal
   [junit4]   2> 315298 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 315298 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard2_replica_n4' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 315299 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51729.solr.core.testSimple1.shard2.replica_n4' (registry 'solr.core.testSimple1.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@383c60a
   [junit4]   2> 315299 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:40572/data
   [junit4]   2> 315299 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 315299 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 315299 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrCore [[testSimple1_shard2_replica_n4] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7DD583F7A5B89E45-001/tempDir-001/node3/testSimple1_shard2_replica_n4], dataDir=[hdfs://localhost:40572/data/testSimple1/core_node7/data/]
   [junit4]   2> 315304 INFO  (qtp1950758694-3240) [n:127.0.0.1:51729_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 315304 INFO  (qtp1950758694-3240) [n:127.0.0.1:51729_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard1_replica_n1' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 315305 INFO  (qtp1950758694-3240) [n:127.0.0.1:51729_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51729.solr.core.testSimple1.shard1.replica_n1' (registry 'solr.core.testSimple1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@383c60a
   [junit4]   2> 315305 INFO  (qtp1950758694-3240) [n:127.0.0.1:51729_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:40572/data
   [junit4]   2> 315305 INFO  (qtp1950758694-3240) [n:127.0.0.1:51729_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 315305 INFO  (qtp1950758694-3240) [n:127.0.0.1:51729_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 315305 INFO  (qtp1950758694-3240) [n:127.0.0.1:51729_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrCore [[testSimple1_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_7DD583F7A5B89E45-001/tempDir-001/node3/testSimple1_shard1_replica_n1], dataDir=[hdfs://localhost:40572/data/testSimple1/core_node3/data/]
   [junit4]   2> 315305 WARN  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 315306 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40572/data/testSimple1/core_node7/data/snapshot_metadata
   [junit4]   2> 315307 INFO  (qtp1950758694-3240) [n:127.0.0.1:51729_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40572/data/testSimple1/core_node3/data/snapshot_metadata
   [junit4]   2> 315309 WARN  (qtp668097388-3254) [n:127.0.0.1:45035_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 315318 WARN  (qtp1950758694-3240) [n:127.0.0.1:51729_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 315325 WARN  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 316187 DEBUG (ScheduledTrigger-1412-thread-3) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 317188 DEBUG (ScheduledTrigger-1412-thread-1) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 318188 DEBUG (ScheduledTrigger-1412-thread-1) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 319188 DEBUG (ScheduledTrigger-1412-thread-4) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 320188 DEBUG (ScheduledTrigger-1412-thread-2) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 320307 WARN  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 320311 WARN  (qtp668097388-3254) [n:127.0.0.1:45035_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 320319 WARN  (qtp1950758694-3240) [n:127.0.0.1:51729_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 320326 WARN  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 321189 DEBUG (ScheduledTrigger-1412-thread-1) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 322189 DEBUG (ScheduledTrigger-1412-thread-1) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 323189 DEBUG (ScheduledTrigger-1412-thread-3) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 324190 DEBUG (ScheduledTrigger-1412-thread-4) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 325190 DEBUG (ScheduledTrigger-1412-thread-3) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 325328 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40572/data/testSimple1/core_node5/data
   [junit4]   2> 325342 INFO  (qtp668097388-3254) [n:127.0.0.1:45035_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40572/data/testSimple1/core_node8/data
   [junit4]   2> 325355 INFO  (qtp1950758694-3240) [n:127.0.0.1:51729_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40572/data/testSimple1/core_node3/data
   [junit4]   2> 325357 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40572/data/testSimple1/core_node7/data
   [junit4]   2> 325396 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40572/data/testSimple1/core_node5/data/index
   [junit4]   2> 325406 INFO  (qtp668097388-3254) [n:127.0.0.1:45035_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40572/data/testSimple1/core_node8/data/index
   [junit4]   2> 325413 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40572/data/testSimple1/core_node7/data/index
   [junit4]   2> 325415 INFO  (qtp1950758694-3240) [n:127.0.0.1:51729_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40572/data/testSimple1/core_node3/data/index
   [junit4]   2> 325909 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51757 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-47ed794c-64fa-46f0-806c-d6f6fd458d50:NORMAL:127.0.0.1:47863|RBW], ReplicaUC[[DISK]DS-77938411-5f42-4694-89d7-7f77f44ee498:NORMAL:127.0.0.1:51757|FINALIZED]]} size 0
   [junit4]   2> 325915 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:47863 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-47ed794c-64fa-46f0-806c-d6f6fd458d50:NORMAL:127.0.0.1:47863|RBW], ReplicaUC[[DISK]DS-77938411-5f42-4694-89d7-7f77f44ee498:NORMAL:127.0.0.1:51757|FINALIZED]]} size 0
   [junit4]   2> 325929 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:47863 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a925587e-2a4d-4721-8240-e45db73c015a:NORMAL:127.0.0.1:47863|RBW], ReplicaUC[[DISK]DS-77938411-5f42-4694-89d7-7f77f44ee498:NORMAL:127.0.0.1:51757|RBW]]} size 69
   [junit4]   2> 325930 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:47863 is added to blk_1073741827_1003{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a925587e-2a4d-4721-8240-e45db73c015a:NORMAL:127.0.0.1:47863|RBW], ReplicaUC[[DISK]DS-cbc87cc0-a176-474c-9082-fe115cd6bb89:NORMAL:127.0.0.1:51757|RBW]]} size 69
   [junit4]   2> 325931 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:47863 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-47ed794c-64fa-46f0-806c-d6f6fd458d50:NORMAL:127.0.0.1:47863|RBW], ReplicaUC[[DISK]DS-cbc87cc0-a176-474c-9082-fe115cd6bb89:NORMAL:127.0.0.1:51757|RBW]]} size 0
   [junit4]   2> 325954 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51757 is added to blk_1073741825_1001 size 69
   [junit4]   2> 325954 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51757 is added to blk_1073741826_1002 size 69
   [junit4]   2> 325954 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51757 is added to blk_1073741827_1003 size 69
   [junit4]   2> 326092 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 326092 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 326092 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 326102 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 326102 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 326102 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 326116 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 326116 INFO  (qtp1950758694-3241) [n:127.0.0.1:51729_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 326117 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 326117 INFO  (qtp668097388-3255) [n:127.0.0.1:45035_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 326190 DEBUG (ScheduledTrigger-1412-thread-4) [n:127.0.0.1:45035_solr    ] o.a.s.c.a.NodeLostTrigger R

[...truncated too long message...]

1:60434_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 2448071 DEBUG (jetty-closer-8106-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 2448074 INFO  (jetty-closer-8106-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@41009a89{/solr,null,UNAVAILABLE}
   [junit4]   2> 2448075 INFO  (jetty-closer-8106-thread-2) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 2448076 ERROR (SUITE-AutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2448076 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[7DD583F7A5B89E45]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50592 50592
   [junit4]   2> 2448245 INFO  (Thread-7387) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50592 50592
   [junit4]   2> 2448246 WARN  (Thread-7387) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	7	/solr/aliases.json
   [junit4]   2> 	5	/solr/collections/testSimple2/terms/shard2
   [junit4]   2> 	5	/solr/collections/testSimple1/terms/shard2
   [junit4]   2> 	5	/solr/collections/testSimple2/terms/shard1
   [junit4]   2> 	5	/solr/collections/testSimple1/terms/shard1
   [junit4]   2> 	3	/solr/security.json
   [junit4]   2> 	3	/solr/configs/conf
   [junit4]   2> 	2	/solr/collections/testSimple2/collectionprops.json
   [junit4]   2> 	2	/solr/collections/testSimple1/collectionprops.json
   [junit4]   2> 	2	/solr/collections/testSimple3/terms/shard3
   [junit4]   2> 	2	/solr/collections/testSimple3/terms/shard2
   [junit4]   2> 	2	/solr/collections/testSimple3/terms/shard1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	17	/solr/collections/testSimple1/state.json
   [junit4]   2> 	14	/solr/collections/testSimple2/state.json
   [junit4]   2> 	9	/solr/collections/testSimple3/state.json
   [junit4]   2> 	7	/solr/clusterprops.json
   [junit4]   2> 	7	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/overseer_elect/election/73442906598408199-127.0.0.1:60434_solr-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	7	/solr/live_nodes
   [junit4]   2> 	7	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_7DD583F7A5B89E45-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1020, maxMBSortInHeap=7.960129243749989, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@1eeb8f0d), locale=th-TH, timezone=America/Manaus
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=194090776,total=520617984
   [junit4]   2> NOTE: All tests run in this JVM: [ForceLeaderTest, MoveReplicaTest, TestSweetSpotSimilarityFactory, BasicZkTest, FastVectorHighlighterTest, TestTestInjection, TestXIncludeConfig, TestNumericTerms32, TestOnReconnectListenerSupport, TestQueryUtils, TestManagedResourceStorage, PolyFieldTest, TestObjectReleaseTracker, TestCollectionAPIs, SampleTest, TestFieldTypeResource, ShardRoutingCustomTest, CloudExitableDirectoryReaderTest, SolrCoreMetricManagerTest, DOMUtilTest, TestSolrJ, TestSkipOverseerOperations, TestSortableTextField, CursorMarkTest, TriLevelCompositeIdRoutingTest, TestFieldCollectionResource, AutoAddReplicasPlanActionTest, TestConfigReload, LoggingHandlerTest, FileUtilsTest, ShufflingReplicaListTransformerTest, AlternateDirectoryTest, TestPostingsSolrHighlighter, TimeRoutedAliasUpdateProcessorTest, TestSimpleTrackingShardHandler, TestDistribDocBasedVersion, TestImplicitCoreProperties, ConnectionManagerTest, BlockDirectoryTest, QueryEqualityTest, TestPointFields, SolrJmxReporterCloudTest, MissingSegmentRecoveryTest, TestFieldCacheReopen, AssignBackwardCompatibilityTest, TestExecutePlanAction, MetricsHandlerTest, TestDocBasedVersionConstraints, TestRecovery, TestRawResponseWriter, HdfsChaosMonkeyNothingIsSafeTest, DistanceUnitsTest, UpdateRequestProcessorFactoryTest, SyncSliceTest, TestRandomCollapseQParserPlugin, PreAnalyzedFieldManagedSchemaCloudTest, TestSolrCloudSnapshots, TestUniqueKeyFieldResource, TestSimpleTextCodec, TestCharFilters, DistributedIntervalFacetingTest, TestSolrCloudWithHadoopAuthPlugin, LeaderVoteWaitTimeoutTest, TestExactSharedStatsCache, TestStressVersions, TestFuzzyAnalyzedSuggestions, TestClusterProperties, ConfigureRecoveryStrategyTest, TestSegmentSorting, CloudMLTQParserTest, AddBlockUpdateTest, SystemInfoHandlerTest, SolrInfoBeanTest, CheckHdfsIndexTest, TestReplicationHandlerBackup, TestSystemIdResolver, BasicFunctionalityTest, TestSchemaSimilarityResource, BigEndianAscendingWordDeserializerTest, TestCopyFieldCollectionResource, SubstringBytesRefFilterTest, TestFoldingMultitermQuery, TestRTGBase, TestElisionMultitermQuery, TestManagedSynonymGraphFilterFactory, TestSimDistributedQueue, TestXmlQParserPlugin, BlobRepositoryCloudTest, TestDistributedGrouping, TestNumericRangeQuery32, BasicAuthIntegrationTest, TestFreeTextSuggestions, TestSolrQueryResponse, TestPullReplicaErrorHandling, JavabinLoaderTest, DistributedMLTComponentTest, TestManagedSchema, DataDrivenBlockJoinTest, TestMultiWordSynonyms, ResponseBuilderTest, DistributedQueueTest, TestLegacyNumericUtils, SoftAutoCommitTest, SolrCLIZkUtilsTest, BadComponentTest, SortByFunctionTest, HDFSCollectionsAPITest, SolrMetricsIntegrationTest, DirectUpdateHandlerOptimizeTest, HttpPartitionTest, TestSurroundQueryParser, TestZkChroot, TestSolrConfigHandlerConcurrent, TestReversedWildcardFilterFactory, TestFastLRUCache, TestHttpShardHandlerFactory, TestAuthenticationFramework, TestSlowCompositeReaderWrapper, LeaderElectionContextKeyTest, VersionInfoTest, HLLUtilTest, AtomicUpdatesTest, ActionThrottleTest, TestCustomStream, RequiredFieldsTest, TestLegacyNumericRangeQueryBuilder, TestBM25SimilarityFactory, SchemaVersionSpecificBehaviorTest, TestStressLiveNodes, ChaosMonkeySafeLeaderTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, ShardRoutingTest, ClusterStateUpdateTest, LeaderElectionTest, ZkSolrClientTest, TestHashPartitioner, DistributedSpellCheckComponentTest, ZkControllerTest, TestStressReorder, TestJoin, TestRangeQuery, DirectUpdateHandlerTest, TestIndexSearcher, DistributedQueryElevationComponentTest, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, SuggesterTSTTest, TestStressLucene, TestTrie, TestCSVLoader, TestUpdate, TestWordDelimiterFilterFactory, XsltUpdateRequestHandlerTest, CacheHeaderTest, TermsComponentTest, FieldAnalysisRequestHandlerTest, SolrPluginUtilsTest, QueryParsingTest, JsonLoaderTest, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, PingRequestHandlerTest, CSVRequestHandlerTest, TestComponentsName, TestLFUCache, TestUtils, SliceStateTest, UUIDFieldTest, TestDistributedMissingSort, CleanupOldIndexTest, CollectionPropsTest, ConfigSetsAPITest, CreateCollectionCleanupTest, DeleteInactiveReplicaTest, DeleteNodeTest, DistribCursorPagingTest, HealthCheckHandlerTest, LeaderFailoverAfterPartitionTest, MoveReplicaHDFSTest, TestCloudConsistency, TestConfigSetsAPI, TestLeaderElectionZkExpiry, TestRandomRequestDistribution, TestSolrCloudWithKerberosAlt, TestTolerantUpdateProcessorCloud, ZkFailoverTest, TestCollectionAPI, TestCollectionsAPIViaSolrCloudCluster, AutoAddReplicasIntegrationTest]
   [junit4] Completed [593/785 (5!)] on J1 in 101.02s, 1 test, 1 failure <<< FAILURES!

[...truncated 8134 lines...]
   [junit4] Suite: org.apache.solr.ltr.TestLTRReRankingPipeline
   [junit4]   2> 37649 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[5394B1BAA58B6095]) [    ] o.a.s.l.TestLTRReRankingPipeline rerank 1 documents 
   [junit4]   2> 37650 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[5394B1BAA58B6095]) [    ] o.a.s.l.TestLTRReRankingPipeline doc 0 in pos 0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTRReRankingPipeline -Dtests.method=testDifferentTopN -Dtests.seed=5394B1BAA58B6095 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ar-QA -Dtests.timezone=America/New_York -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.10s J0 | TestLTRReRankingPipeline.testDifferentTopN <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1.0> but was:<0.0>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([5394B1BAA58B6095:A235C3EA9030AA07]:0)
   [junit4]    > 	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {field=FSTOrd50, id=Lucene50(blocksize=128), final-score=Lucene50(blocksize=128)}, docValues:{final-score=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=1132, maxMBSortInHeap=7.8683016552938, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@58ce60fe), locale=ar-QA, timezone=America/New_York
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=199489264,total=311427072
   [junit4]   2> NOTE: All tests run in this JVM: [TestLTROnSolrCloud, TestFieldLengthFeature, TestModelManagerPersistence, TestUserTermScorereQDF, TestLTRReRankingPipeline]
   [junit4] Completed [35/37 (1!)] on J0 in 0.19s, 3 tests, 1 failure <<< FAILURES!

[...truncated 41951 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 11 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/11/

6 tests failed.
FAILED:  org.apache.solr.core.TestDynamicLoading.testDynamicLoading

Error Message:
Error from server at https://127.0.0.1:37117: At least one of the node(s) specified [127.0.0.1:37327_] are not currently active in [], no action taken.

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:37117: At least one of the node(s) specified [127.0.0.1:37327_] are not currently active in [], no action taken.
	at __randomizedtesting.SeedInfo.seed([FF0A05E04BCD766F:274728B7BC10D3CF]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1105)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:885)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:818)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:425)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:341)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestDynamicLoading

Error Message:
42 threads leaked from SUITE scope at org.apache.solr.core.TestDynamicLoading:     1) Thread[id=13764, name=qtp2060837204-13764, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=13774, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=13728, name=qtp1239436406-13728-acceptor-0@4c8cc436-ServerConnector@4e51f027{SSL,[ssl, http/1.1]}{127.0.0.1:40515}, state=RUNNABLE, group=TGRP-TestDynamicLoading]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:379)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:638)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=13773, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=13730, name=qtp1239436406-13730, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=13739, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=13744, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-EventThread, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)    8) Thread[id=13732, name=qtp1239436406-13732, state=RUNNABLE, group=TGRP-TestDynamicLoading]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)    9) Thread[id=13791, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-EventThread, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   10) Thread[id=13761, name=qtp2060837204-13761, state=RUNNABLE, group=TGRP-TestDynamicLoading]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.io.ManagedSelector$$Lambda$230/2024484628.run(Unknown Source)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   11) Thread[id=13790, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)   12) Thread[id=13760, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   13) Thread[id=13727, name=qtp1239436406-13727, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   14) Thread[id=13755, name=Scheduler-274641482, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   15) Thread[id=13785, name=zkConnectionManagerCallback-2889-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   16) Thread[id=13800, name=zkConnectionManagerCallback-2881-thread-1-processing-n:127.0.0.1:40515_-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)   17) Thread[id=13768, name=qtp2060837204-13768, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   18) Thread[id=13783, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)   19) Thread[id=13766, name=qtp2060837204-13766, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   20) Thread[id=13763, name=qtp2060837204-13763-acceptor-0@5c5ae9ec-ServerConnector@473d3b64{SSL,[ssl, http/1.1]}{127.0.0.1:37327}, state=RUNNABLE, group=TGRP-TestDynamicLoading]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:379)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:638)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   21) Thread[id=13753, name=zkConnectionManagerCallback-2881-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   22) Thread[id=13731, name=qtp1239436406-13731, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   23) Thread[id=13743, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)   24) Thread[id=13750, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   25) Thread[id=13769, name=Scheduler-772186916, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   26) Thread[id=13801, name=zkConnectionManagerCallback-2881-thread-1-processing-n:127.0.0.1:40515_-EventThread, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   27) Thread[id=13789, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   28) Thread[id=13758, name=searcherExecutor-4174-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   29) Thread[id=13787, name=Thread-4689, state=WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1731)   30) Thread[id=13767, name=qtp2060837204-13767, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   31) Thread[id=13726, name=qtp1239436406-13726, state=RUNNABLE, group=TGRP-TestDynamicLoading]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.io.ManagedSelector$$Lambda$230/2024484628.run(Unknown Source)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   32) Thread[id=13765, name=qtp2060837204-13765, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   33) Thread[id=13792, name=zkConnectionManagerCallback-2894-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   34) Thread[id=13745, name=zkConnectionManagerCallback-2876-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   35) Thread[id=13734, name=Scheduler-1137720819, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   36) Thread[id=13733, name=qtp1239436406-13733, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)         at java.lang.Thread.run(Thread.java:748)   37) Thread[id=13762, name=qtp2060837204-13762, state=RUNNABLE, group=TGRP-TestDynamicLoading]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.io.ManagedSelector$$Lambda$230/2024484628.run(Unknown Source)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   38) Thread[id=13747, name=zkCallback-2875-thread-1, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   39) Thread[id=13729, name=qtp1239436406-13729, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   40) Thread[id=13784, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-EventThread, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)   41) Thread[id=13738, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   42) Thread[id=13748, name=Thread-4680, state=WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1731)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 42 threads leaked from SUITE scope at org.apache.solr.core.TestDynamicLoading: 
   1) Thread[id=13764, name=qtp2060837204-13764, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=13774, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=13728, name=qtp1239436406-13728-acceptor-0@4c8cc436-ServerConnector@4e51f027{SSL,[ssl, http/1.1]}{127.0.0.1:40515}, state=RUNNABLE, group=TGRP-TestDynamicLoading]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:379)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:638)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=13773, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=13730, name=qtp1239436406-13730, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=13739, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=13744, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-EventThread, state=WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
   8) Thread[id=13732, name=qtp1239436406-13732, state=RUNNABLE, group=TGRP-TestDynamicLoading]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
   9) Thread[id=13791, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-EventThread, state=WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
  10) Thread[id=13761, name=qtp2060837204-13761, state=RUNNABLE, group=TGRP-TestDynamicLoading]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$230/2024484628.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  11) Thread[id=13790, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
  12) Thread[id=13760, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  13) Thread[id=13727, name=qtp1239436406-13727, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)
        at java.lang.Thread.run(Thread.java:748)
  14) Thread[id=13755, name=Scheduler-274641482, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  15) Thread[id=13785, name=zkConnectionManagerCallback-2889-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  16) Thread[id=13800, name=zkConnectionManagerCallback-2881-thread-1-processing-n:127.0.0.1:40515_-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
  17) Thread[id=13768, name=qtp2060837204-13768, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)
        at java.lang.Thread.run(Thread.java:748)
  18) Thread[id=13783, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
  19) Thread[id=13766, name=qtp2060837204-13766, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)
        at java.lang.Thread.run(Thread.java:748)
  20) Thread[id=13763, name=qtp2060837204-13763-acceptor-0@5c5ae9ec-ServerConnector@473d3b64{SSL,[ssl, http/1.1]}{127.0.0.1:37327}, state=RUNNABLE, group=TGRP-TestDynamicLoading]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:379)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:638)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  21) Thread[id=13753, name=zkConnectionManagerCallback-2881-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  22) Thread[id=13731, name=qtp1239436406-13731, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)
        at java.lang.Thread.run(Thread.java:748)
  23) Thread[id=13743, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
  24) Thread[id=13750, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  25) Thread[id=13769, name=Scheduler-772186916, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  26) Thread[id=13801, name=zkConnectionManagerCallback-2881-thread-1-processing-n:127.0.0.1:40515_-EventThread, state=WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
  27) Thread[id=13789, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  28) Thread[id=13758, name=searcherExecutor-4174-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  29) Thread[id=13787, name=Thread-4689, state=WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1731)
  30) Thread[id=13767, name=qtp2060837204-13767, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)
        at java.lang.Thread.run(Thread.java:748)
  31) Thread[id=13726, name=qtp1239436406-13726, state=RUNNABLE, group=TGRP-TestDynamicLoading]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$230/2024484628.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  32) Thread[id=13765, name=qtp2060837204-13765, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)
        at java.lang.Thread.run(Thread.java:748)
  33) Thread[id=13792, name=zkConnectionManagerCallback-2894-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  34) Thread[id=13745, name=zkConnectionManagerCallback-2876-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  35) Thread[id=13734, name=Scheduler-1137720819, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  36) Thread[id=13733, name=qtp1239436406-13733, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:600)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:663)
        at java.lang.Thread.run(Thread.java:748)
  37) Thread[id=13762, name=qtp2060837204-13762, state=RUNNABLE, group=TGRP-TestDynamicLoading]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$230/2024484628.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  38) Thread[id=13747, name=zkCallback-2875-thread-1, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  39) Thread[id=13729, name=qtp1239436406-13729, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  40) Thread[id=13784, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-EventThread, state=WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
  41) Thread[id=13738, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  42) Thread[id=13748, name=Thread-4680, state=WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1731)
	at __randomizedtesting.SeedInfo.seed([FF0A05E04BCD766F]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestDynamicLoading

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=13753, name=zkConnectionManagerCallback-2881-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=13743, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)    3) Thread[id=13769, name=Scheduler-772186916, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=13732, name=qtp1239436406-13732, state=RUNNABLE, group=TGRP-TestDynamicLoading]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=13761, name=qtp2060837204-13761, state=RUNNABLE, group=TGRP-TestDynamicLoading]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.io.ManagedSelector$$Lambda$230/2024484628.run(Unknown Source)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=13758, name=searcherExecutor-4174-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=13790, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)    8) Thread[id=13787, name=Thread-4689, state=WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1731)    9) Thread[id=13755, name=Scheduler-274641482, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   10) Thread[id=13726, name=qtp1239436406-13726, state=RUNNABLE, group=TGRP-TestDynamicLoading]         at sun.nio.ch.IOUtil.drain(Native Method)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:104)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.io.ManagedSelector$$Lambda$230/2024484628.run(Unknown Source)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   11) Thread[id=13792, name=zkConnectionManagerCallback-2894-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   12) Thread[id=13745, name=zkConnectionManagerCallback-2876-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   13) Thread[id=13734, name=Scheduler-1137720819, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   14) Thread[id=13762, name=qtp2060837204-13762, state=RUNNABLE, group=TGRP-TestDynamicLoading]         at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)         at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)         at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:207)         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)         at org.eclipse.jetty.io.ManagedSelector$$Lambda$230/2024484628.run(Unknown Source)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   15) Thread[id=13785, name=zkConnectionManagerCallback-2889-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   16) Thread[id=13800, name=zkConnectionManagerCallback-2881-thread-1-processing-n:127.0.0.1:40515_-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)   17) Thread[id=13747, name=zkCallback-2875-thread-1, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   18) Thread[id=13729, name=qtp1239436406-13729, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   19) Thread[id=13783, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)   20) Thread[id=13748, name=Thread-4680, state=WAITING, group=TGRP-TestDynamicLoading]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1731)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=13753, name=zkConnectionManagerCallback-2881-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=13743, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
   3) Thread[id=13769, name=Scheduler-772186916, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=13732, name=qtp1239436406-13732, state=RUNNABLE, group=TGRP-TestDynamicLoading]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=13761, name=qtp2060837204-13761, state=RUNNABLE, group=TGRP-TestDynamicLoading]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$230/2024484628.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=13758, name=searcherExecutor-4174-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=13790, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
   8) Thread[id=13787, name=Thread-4689, state=WAITING, group=TGRP-TestDynamicLoading]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1731)
   9) Thread[id=13755, name=Scheduler-274641482, state=WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  10) Thread[id=13726, name=qtp1239436406-13726, state=RUNNABLE, group=TGRP-TestDynamicLoading]
        at sun.nio.ch.IOUtil.drain(Native Method)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:104)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
        at org.eclipse.jetty.io.ManagedSelector$$Lambda$230/2024484628.run(Unknown Source)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  11) Thread[id=13792, name=zkConnectionManagerCallback-2894-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  12) Thread[id=13745, name=zkConnectionManagerCallback-2876-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  13) Thread[id=13734, name=Scheduler-1137720819, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  14) Thread[id=13762, name=qtp2060837204-13762, state=RUNNABLE, group=TGRP-TestDynamicLoading]
        at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
        at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
        at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:207)
        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorI

[...truncated too long message...]

java.lang.Thread.run(Thread.java:748)
   [junit4]    >   12) Thread[id=13745, name=zkConnectionManagerCallback-2876-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   13) Thread[id=13734, name=Scheduler-1137720819, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   14) Thread[id=13762, name=qtp2060837204-13762, state=RUNNABLE, group=TGRP-TestDynamicLoading]
   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
   [junit4]    >         at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:207)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
   [junit4]    >         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:375)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:304)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:179)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector$$Lambda$230/2024484628.run(Unknown Source)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   15) Thread[id=13785, name=zkConnectionManagerCallback-2889-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   16) Thread[id=13800, name=zkConnectionManagerCallback-2881-thread-1-processing-n:127.0.0.1:40515_-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
   [junit4]    >   17) Thread[id=13747, name=zkCallback-2875-thread-1, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   18) Thread[id=13729, name=qtp1239436406-13729, state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
   [junit4]    >         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]    >         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   19) Thread[id=13783, name=TEST-TestDynamicLoading.testDynamicLoading-seed#[FF0A05E04BCD766F]-SendThread(127.0.0.1:36848), state=TIMED_WAITING, group=TGRP-TestDynamicLoading]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
   [junit4]    >   20) Thread[id=13748, name=Thread-4680, state=WAITING, group=TGRP-TestDynamicLoading]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:502)
   [junit4]    >         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1731)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([FF0A05E04BCD766F]:0)
   [junit4] Completed [356/785 (3!)] on J0 in 104.16s, 1 test, 3 errors <<< FAILURES!

[...truncated 8814 lines...]
   [junit4] Suite: org.apache.solr.ltr.TestLTRReRankingPipeline
   [junit4]   2> 70748 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[49BD2FF6621E3C37]) [    ] o.a.s.l.TestLTRReRankingPipeline rerank 1 documents 
   [junit4]   2> 70762 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[49BD2FF6621E3C37]) [    ] o.a.s.l.TestLTRReRankingPipeline doc 0 in pos 0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTRReRankingPipeline -Dtests.method=testDifferentTopN -Dtests.seed=49BD2FF6621E3C37 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ru-RU -Dtests.timezone=Africa/Lagos -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.55s J0 | TestLTRReRankingPipeline.testDifferentTopN <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1.0> but was:<0.0>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([49BD2FF6621E3C37:B81C5DA657A5F6A5]:0)
   [junit4]    > 	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@75ab8094), locale=ru-RU, timezone=Africa/Lagos
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=239365592,total=322961408
   [junit4]   2> NOTE: All tests run in this JVM: [TestRankingFeature, TestDefaultWrapperModel, TestNeuralNetworkModel, TestFieldLengthFeature, TestExternalValueFeatures, TestLTRWithSort, TestLTRReRankingPipeline]
   [junit4] Completed [18/37 (1!)] on J0 in 0.98s, 3 tests, 1 failure <<< FAILURES!

[...truncated 41999 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 10 - Still unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/10/

11 tests failed.
FAILED:  org.apache.solr.cloud.TestAuthenticationFramework.testBasics

Error Message:
Error from server at http://127.0.0.1:42367/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/testcollection_shard1_replica_n3/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason: <pre>    Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.8.v20171121</a><hr/>  </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:42367/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.8.v20171121</a><hr/>

</body>
</html>

	at __randomizedtesting.SeedInfo.seed([EC7BE4E50471C09A:D1A34AC93C9F9EEA]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1014)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:885)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:947)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:947)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:947)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:947)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:947)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:818)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
	at org.apache.solr.cloud.TestAuthenticationFramework.collectionCreateSearchDeleteTwice(TestAuthenticationFramework.java:127)
	at org.apache.solr.cloud.TestAuthenticationFramework.testBasics(TestAuthenticationFramework.java:75)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:42367/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.8.v20171121</a><hr/>

</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more


FAILED:  org.apache.solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete

Error Message:
Error from server at http://127.0.0.1:56598/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason: <pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.8.v20171121</a><hr/>  </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:56598/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.8.v20171121</a><hr/>

</body>
</html>

	at __randomizedtesting.SeedInfo.seed([EC7BE4E50471C09A:4F814A4083992A3F]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1014)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:885)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:947)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:947)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:947)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:947)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:947)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:818)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
	at org.apache.solr.cloud.api.collections.TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete(TestCollectionsAPIViaSolrCloudCluster.java:170)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:56598/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.8.v20171121</a><hr/>

</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more


FAILED:  org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:52321_solr, 127.0.0.1:53854_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/8)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "core":"testSimple1_shard1_replica_n1",           "base_url":"https://127.0.0.1:52321/solr",           "node_name":"127.0.0.1:52321_solr",           "state":"active",           "type":"NRT",           "leader":"true"},         "core_node5":{           "core":"testSimple1_shard1_replica_n2",           "base_url":"https://127.0.0.1:59901/solr",           "node_name":"127.0.0.1:59901_solr",           "state":"down",           "type":"NRT"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node7":{           "core":"testSimple1_shard2_replica_n4",           "base_url":"https://127.0.0.1:52321/solr",           "node_name":"127.0.0.1:52321_solr",           "state":"active",           "type":"NRT",           "leader":"true"},         "core_node8":{           "core":"testSimple1_shard2_replica_n6",           "base_url":"https://127.0.0.1:59901/solr",           "node_name":"127.0.0.1:59901_solr",           "state":"down",           "type":"NRT"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:52321_solr, 127.0.0.1:53854_solr]
Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/8)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "core":"testSimple1_shard1_replica_n1",
          "base_url":"https://127.0.0.1:52321/solr",
          "node_name":"127.0.0.1:52321_solr",
          "state":"active",
          "type":"NRT",
          "leader":"true"},
        "core_node5":{
          "core":"testSimple1_shard1_replica_n2",
          "base_url":"https://127.0.0.1:59901/solr",
          "node_name":"127.0.0.1:59901_solr",
          "state":"down",
          "type":"NRT"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "core":"testSimple1_shard2_replica_n4",
          "base_url":"https://127.0.0.1:52321/solr",
          "node_name":"127.0.0.1:52321_solr",
          "state":"active",
          "type":"NRT",
          "leader":"true"},
        "core_node8":{
          "core":"testSimple1_shard2_replica_n6",
          "base_url":"https://127.0.0.1:59901/solr",
          "node_name":"127.0.0.1:59901_solr",
          "state":"down",
          "type":"NRT"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
	at __randomizedtesting.SeedInfo.seed([EC7BE4E50471C09A:D4C8C01B2382144B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:269)
	at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:94)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:49577_solr, 127.0.0.1:52510_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/9)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "dataDir":"hdfs://localhost:44954/data/testSimple1/core_node3/data/",           "base_url":"http://127.0.0.1:45952/solr",           "node_name":"127.0.0.1:45952_solr",           "type":"NRT",           "ulogDir":"hdfs://localhost:44954/data/testSimple1/core_node3/data/tlog",           "core":"testSimple1_shard1_replica_n1",           "shared_storage":"true",           "state":"down"},         "core_node5":{           "dataDir":"hdfs://localhost:44954/data/testSimple1/core_node5/data/",           "base_url":"http://127.0.0.1:49577/solr",           "node_name":"127.0.0.1:49577_solr",           "type":"NRT",           "ulogDir":"hdfs://localhost:44954/data/testSimple1/core_node5/data/tlog",           "core":"testSimple1_shard1_replica_n2",           "shared_storage":"true",           "state":"active",           "leader":"true"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node7":{           "dataDir":"hdfs://localhost:44954/data/testSimple1/core_node7/data/",           "base_url":"http://127.0.0.1:45952/solr",           "node_name":"127.0.0.1:45952_solr",           "type":"NRT",           "ulogDir":"hdfs://localhost:44954/data/testSimple1/core_node7/data/tlog",           "core":"testSimple1_shard2_replica_n4",           "shared_storage":"true",           "state":"down"},         "core_node8":{           "dataDir":"hdfs://localhost:44954/data/testSimple1/core_node8/data/",           "base_url":"http://127.0.0.1:49577/solr",           "node_name":"127.0.0.1:49577_solr",           "type":"NRT",           "ulogDir":"hdfs://localhost:44954/data/testSimple1/core_node8/data/tlog",           "core":"testSimple1_shard2_replica_n6",           "shared_storage":"true",           "state":"active",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:49577_solr, 127.0.0.1:52510_solr]
Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/9)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://localhost:44954/data/testSimple1/core_node3/data/",
          "base_url":"http://127.0.0.1:45952/solr",
          "node_name":"127.0.0.1:45952_solr",
          "type":"NRT",
          "ulogDir":"hdfs://localhost:44954/data/testSimple1/core_node3/data/tlog",
          "core":"testSimple1_shard1_replica_n1",
          "shared_storage":"true",
          "state":"down"},
        "core_node5":{
          "dataDir":"hdfs://localhost:44954/data/testSimple1/core_node5/data/",
          "base_url":"http://127.0.0.1:49577/solr",
          "node_name":"127.0.0.1:49577_solr",
          "type":"NRT",
          "ulogDir":"hdfs://localhost:44954/data/testSimple1/core_node5/data/tlog",
          "core":"testSimple1_shard1_replica_n2",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://localhost:44954/data/testSimple1/core_node7/data/",
          "base_url":"http://127.0.0.1:45952/solr",
          "node_name":"127.0.0.1:45952_solr",
          "type":"NRT",
          "ulogDir":"hdfs://localhost:44954/data/testSimple1/core_node7/data/tlog",
          "core":"testSimple1_shard2_replica_n4",
          "shared_storage":"true",
          "state":"down"},
        "core_node8":{
          "dataDir":"hdfs://localhost:44954/data/testSimple1/core_node8/data/",
          "base_url":"http://127.0.0.1:49577/solr",
          "node_name":"127.0.0.1:49577_solr",
          "type":"NRT",
          "ulogDir":"hdfs://localhost:44954/data/testSimple1/core_node8/data/tlog",
          "core":"testSimple1_shard2_replica_n6",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
	at __randomizedtesting.SeedInfo.seed([EC7BE4E50471C09A:D4C8C01B2382144B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:269)
	at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:94)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue.testDistributedQueueBlocking

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
	at __randomizedtesting.SeedInfo.seed([EC7BE4E50471C09A:A9D1969C40237CEE]:0)
	at java.util.concurrent.FutureTask.get(FutureTask.java:205)
	at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.testDistributedQueueBlocking(TestSimDistributedQueue.java:101)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue:     1) Thread[id=334, name=sdqtest--11-thread-1, state=TIMED_WAITING, group=TGRP-TestGenericDistributedQueue]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:194)         at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:167)         at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.lambda$testDistributedQueueBlocking$0(TestSimDistributedQueue.java:92)         at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue$$Lambda$262/1386854269.call(Unknown Source)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$125/621366843.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue: 
   1) Thread[id=334, name=sdqtest--11-thread-1, state=TIMED_WAITING, group=TGRP-TestGenericDistributedQueue]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:194)
        at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:167)
        at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.lambda$testDistributedQueueBlocking$0(TestSimDistributedQueue.java:92)
        at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue$$Lambda$262/1386854269.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$125/621366843.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([EC7BE4E50471C09A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestLargeCluster

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster:     1) Thread[id=24944, name=AutoscalingActionExecutor-8716-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:131)         at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:110)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)         at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:108)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:74)         at org.apache.solr.client.solrj.cloud.autoscaling.Row.copy(Row.java:91)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.lambda$getMatrixCopy$1(Policy.java:297)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session$$Lambda$584/1971910727.apply(Unknown Source)         at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)         at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:298)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:287)         at org.apache.solr.client.solrj.cloud.autoscaling.Row.addReplica(Row.java:122)         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:59)         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)         at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)         at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$397/793967629.run(Unknown Source)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$119/1716245607.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster: 
   1) Thread[id=24944, name=AutoscalingActionExecutor-8716-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:131)
        at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:110)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)
        at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:108)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:74)
        at org.apache.solr.client.solrj.cloud.autoscaling.Row.copy(Row.java:91)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.lambda$getMatrixCopy$1(Policy.java:297)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session$$Lambda$584/1971910727.apply(Unknown Source)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:298)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:287)
        at org.apache.solr.client.solrj.cloud.autoscaling.Row.addReplica(Row.java:122)
        at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:59)
        at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)
        at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)
        at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$397/793967629.run(Unknown Source)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$119/1716245607.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([EC7BE4E50471C09A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestLargeCluster

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=24944, name=AutoscalingActionExecutor-8716-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]         at java.util.ArrayList.iterator(ArrayList.java:834)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:131)         at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:110)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)         at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:108)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:74)         at org.apache.solr.client.solrj.cloud.autoscaling.Row.copy(Row.java:91)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.lambda$getMatrixCopy$1(Policy.java:297)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session$$Lambda$584/1971910727.apply(Unknown Source)         at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)         at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:298)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:287)         at org.apache.solr.client.solrj.cloud.autoscaling.Row.removeReplica(Row.java:156)         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:60)         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)         at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)         at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$397/793967629.run(Unknown Source)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$119/1716245607.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=24944, name=AutoscalingActionExecutor-8716-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]
        at java.util.ArrayList.iterator(ArrayList.java:834)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:131)
        at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:110)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)
        at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:108)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:74)
        at org.apache.solr.client.solrj.cloud.autoscaling.Row.copy(Row.java:91)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.lambda$getMatrixCopy$1(Policy.java:297)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session$$Lambda$584/1971910727.apply(Unknown Source)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:298)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:287)
        at org.apache.solr.client.solrj.cloud.autoscaling.Row.removeReplica(Row.java:156)
        at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:60)
        at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)
        at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)
        at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$397/793967629.run(Unknown Source)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$119/1716245607.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([EC7BE4E50471C09A]:0)


FAILED:  org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN

Error Message:
expected:<1.0> but was:<0.0>

Stack Trace:
java.lang.AssertionError: expected:<1.0> but was:<0.0>
	at __randomizedtesting.SeedInfo.seed([75AA0CB61F054D50:840B7EE62ABE87C2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:443)
	at org.junit.Assert.assertEquals(Assert.java:512)
	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload

Error Message:
Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>

Stack Trace:
java.lang.AssertionError: Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>
	at __randomizedtesting.SeedInfo.seed([EC7BE4E50471C09A:EEC23F8977321C57]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload(TestJmxIntegration.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart

Error Message:
IOException occured when talking to server at: http://127.0.0.1:48678/solr/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:48678/solr/collection1
	at __randomizedtesting.SeedInfo.seed([EC7BE4E50471C09A:348C2001AFAA02C6]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
	at org.apache.solr.handler.TestReplicationHandler.index(TestReplicationHandler.java:180)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart(TestReplicationHandler.java:643)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:48678 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 47 more




Build Log:
[...truncated 12070 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestAuthenticationFramework
   [junit4]   2> 27261 INFO  (SUITE-TestAuthenticationFramework-seed#[EC7BE4E50471C09A]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_EC7BE4E50471C09A-001/init-core-data-001
   [junit4]   2> 27262 WARN  (SUITE-TestAuthenticationFramework-seed#[EC7BE4E50471C09A]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=7 numCloses=7
   [junit4]   2> 27263 INFO  (SUITE-TestAuthenticationFramework-seed#[EC7BE4E50471C09A]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 27265 INFO  (SUITE-TestAuthenticationFramework-seed#[EC7BE4E50471C09A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 27314 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[EC7BE4E50471C09A]) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_EC7BE4E50471C09A-001/tempDir-001
   [junit4]   2> 27317 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[EC7BE4E50471C09A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 27319 INFO  (Thread-11) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 27319 INFO  (Thread-11) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 27394 ERROR (Thread-11) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 27419 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[EC7BE4E50471C09A]) [    ] o.a.s.c.ZkTestServer start zk server on port:58139
   [junit4]   2> 27448 INFO  (zkConnectionManagerCallback-27-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 28042 INFO  (jetty-launcher-24-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 28047 INFO  (jetty-launcher-24-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 28172 INFO  (jetty-launcher-24-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 28172 INFO  (jetty-launcher-24-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 28176 INFO  (jetty-launcher-24-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 28251 INFO  (jetty-launcher-24-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 28263 INFO  (jetty-launcher-24-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 28263 INFO  (jetty-launcher-24-thread-1) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 28285 INFO  (jetty-launcher-24-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@535b1f49{/solr,null,AVAILABLE}
   [junit4]   2> 28303 INFO  (jetty-launcher-24-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@135e75cf{/solr,null,AVAILABLE}
   [junit4]   2> 28325 INFO  (jetty-launcher-24-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@79f47188{HTTP/1.1,[http/1.1]}{127.0.0.1:42367}
   [junit4]   2> 28325 INFO  (jetty-launcher-24-thread-1) [    ] o.e.j.s.Server Started @31206ms
   [junit4]   2> 28325 INFO  (jetty-launcher-24-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42367}
   [junit4]   2> 28336 INFO  (jetty-launcher-24-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@242b4b73{HTTP/1.1,[http/1.1]}{127.0.0.1:59601}
   [junit4]   2> 28348 INFO  (jetty-launcher-24-thread-2) [    ] o.e.j.s.Server Started @31229ms
   [junit4]   2> 28348 INFO  (jetty-launcher-24-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59601}
   [junit4]   2> 28362 ERROR (jetty-launcher-24-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 28362 INFO  (jetty-launcher-24-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 28362 INFO  (jetty-launcher-24-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 28371 INFO  (jetty-launcher-24-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 28363 ERROR (jetty-launcher-24-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 28379 INFO  (jetty-launcher-24-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 28379 INFO  (jetty-launcher-24-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 28381 INFO  (jetty-launcher-24-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 28427 INFO  (jetty-launcher-24-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-10T21:42:23.957Z
   [junit4]   2> 28429 INFO  (jetty-launcher-24-thread-2) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 28451 INFO  (jetty-launcher-24-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-10T21:42:23.953Z
   [junit4]   2> 28491 INFO  (zkConnectionManagerCallback-29-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 28499 INFO  (zkConnectionManagerCallback-31-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 28504 INFO  (jetty-launcher-24-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 28522 INFO  (jetty-launcher-24-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 28769 INFO  (jetty-launcher-24-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58139/solr
   [junit4]   2> 28787 INFO  (jetty-launcher-24-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58139/solr
   [junit4]   2> 28897 INFO  (zkConnectionManagerCallback-39-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 28963 INFO  (zkConnectionManagerCallback-38-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 28984 INFO  (zkConnectionManagerCallback-41-thread-1-processing-n:127.0.0.1:59601_solr) [n:127.0.0.1:59601_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 29123 INFO  (zkConnectionManagerCallback-43-thread-1-processing-n:127.0.0.1:42367_solr) [n:127.0.0.1:42367_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 29582 INFO  (jetty-launcher-24-thread-1) [n:127.0.0.1:42367_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 29584 INFO  (jetty-launcher-24-thread-1) [n:127.0.0.1:42367_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42367_solr
   [junit4]   2> 29586 INFO  (jetty-launcher-24-thread-1) [n:127.0.0.1:42367_solr    ] o.a.s.c.Overseer Overseer (id=73431886940864518-127.0.0.1:42367_solr-n_0000000000) starting
   [junit4]   2> 29591 INFO  (jetty-launcher-24-thread-2) [n:127.0.0.1:59601_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 29596 INFO  (jetty-launcher-24-thread-2) [n:127.0.0.1:59601_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59601_solr
   [junit4]   2> 29664 INFO  (zkCallback-42-thread-1-processing-n:127.0.0.1:42367_solr) [n:127.0.0.1:42367_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 29665 INFO  (zkCallback-40-thread-1-processing-n:127.0.0.1:59601_solr) [n:127.0.0.1:59601_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 30001 INFO  (jetty-launcher-24-thread-2) [n:127.0.0.1:59601_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin
   [junit4]   2> 30161 INFO  (zkConnectionManagerCallback-49-thread-1-processing-n:127.0.0.1:42367_solr) [n:127.0.0.1:42367_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 30173 INFO  (jetty-launcher-24-thread-1) [n:127.0.0.1:42367_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 30175 INFO  (jetty-launcher-24-thread-1) [n:127.0.0.1:42367_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58139/solr ready
   [junit4]   2> 30467 INFO  (jetty-launcher-24-thread-2) [n:127.0.0.1:59601_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59601.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5aac74c1
   [junit4]   2> 30468 INFO  (jetty-launcher-24-thread-2) [n:127.0.0.1:59601_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59601.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5aac74c1
   [junit4]   2> 30468 INFO  (jetty-launcher-24-thread-2) [n:127.0.0.1:59601_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59601.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5aac74c1
   [junit4]   2> 30472 INFO  (jetty-launcher-24-thread-2) [n:127.0.0.1:59601_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_EC7BE4E50471C09A-001/tempDir-001/node2/.
   [junit4]   2> 30481 INFO  (jetty-launcher-24-thread-1) [n:127.0.0.1:42367_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42367_solr
   [junit4]   2> 30538 INFO  (zkCallback-42-thread-2-processing-n:127.0.0.1:42367_solr) [n:127.0.0.1:42367_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 30620 INFO  (zkCallback-40-thread-1-processing-n:127.0.0.1:59601_solr) [n:127.0.0.1:59601_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 30631 INFO  (zkCallback-48-thread-1-processing-n:127.0.0.1:42367_solr) [n:127.0.0.1:42367_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 30639 INFO  (zkConnectionManagerCallback-53-thread-1-processing-n:127.0.0.1:59601_solr) [n:127.0.0.1:59601_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 30664 INFO  (jetty-launcher-24-thread-2) [n:127.0.0.1:59601_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 30666 INFO  (jetty-launcher-24-thread-2) [n:127.0.0.1:59601_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58139/solr ready
   [junit4]   2> 30847 INFO  (jetty-launcher-24-thread-1) [n:127.0.0.1:42367_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin
   [junit4]   2> 30947 INFO  (jetty-launcher-24-thread-1) [n:127.0.0.1:42367_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42367.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5aac74c1
   [junit4]   2> 30994 INFO  (jetty-launcher-24-thread-1) [n:127.0.0.1:42367_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42367.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5aac74c1
   [junit4]   2> 30995 INFO  (jetty-launcher-24-thread-1) [n:127.0.0.1:42367_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42367.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5aac74c1
   [junit4]   2> 31015 INFO  (jetty-launcher-24-thread-1) [n:127.0.0.1:42367_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_EC7BE4E50471C09A-001/tempDir-001/node1/.
   [junit4]   2> 31403 INFO  (zkConnectionManagerCallback-55-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 31466 INFO  (zkConnectionManagerCallback-59-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 31472 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[EC7BE4E50471C09A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 31475 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[EC7BE4E50471C09A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58139/solr ready
   [junit4]   2> 31481 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[EC7BE4E50471C09A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testBasics
   [junit4]   2> 32061 INFO  (qtp1705980688-107) [n:127.0.0.1:59601_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params async=195bbadd-9c5c-4462-a990-d44a26774904&replicationFactor=2&collection.configName=solrCloudCollectionConfig&maxShardsPerNode=2&name=testcollection&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 32092 INFO  (qtp1705980688-107) [n:127.0.0.1:59601_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={async=195bbadd-9c5c-4462-a990-d44a26774904&replicationFactor=2&collection.configName=solrCloudCollectionConfig&maxShardsPerNode=2&name=testcollection&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=37
   [junit4]   2> 32120 INFO  (OverseerThreadFactory-26-thread-1-processing-n:127.0.0.1:42367_solr) [n:127.0.0.1:42367_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testcollection
   [junit4]   2> 32377 INFO  (qtp1705980688-109) [n:127.0.0.1:59601_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=195bbadd-9c5c-4462-a990-d44a26774904&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 32387 INFO  (qtp1705980688-109) [n:127.0.0.1:59601_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=195bbadd-9c5c-4462-a990-d44a26774904&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=10
   [junit4]   2> 32432 INFO  (OverseerStateUpdate-73431886940864518-127.0.0.1:42367_solr-n_0000000000) [n:127.0.0.1:42367_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testcollection_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:59601/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 32444 INFO  (OverseerStateUpdate-73431886940864518-127.0.0.1:42367_solr-n_0000000000) [n:127.0.0.1:42367_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testcollection_shard1_replica_n3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:42367/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 32451 INFO  (OverseerStateUpdate-73431886940864518-127.0.0.1:42367_solr-n_0000000000) [n:127.0.0.1:42367_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testcollection_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:59601/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 32459 INFO  (OverseerStateUpdate-73431886940864518-127.0.0.1:42367_solr-n_0000000000) [n:127.0.0.1:42367_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testcollection_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:42367/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 32819 INFO  (qtp1705980688-109) [n:127.0.0.1:59601_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=195bbadd-9c5c-4462-a990-d44a2677490420970050602881683&coreNodeName=core_node2&name=testcollection_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=0 QTime=6
   [junit4]   2> 32821 INFO  (parallelCoreAdminExecutor-22-thread-1-processing-n:127.0.0.1:59601_solr 195bbadd-9c5c-4462-a990-d44a2677490420970050602881683 CREATE) [n:127.0.0.1:59601_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=195bbadd-9c5c-4462-a990-d44a2677490420970050602881683&coreNodeName=core_node2&name=testcollection_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 32836 INFO  (qtp2006433491-115) [n:127.0.0.1:42367_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=195bbadd-9c5c-4462-a990-d44a2677490420970050616037112&coreNodeName=core_node5&name=testcollection_shard1_replica_n3&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=0 QTime=12
   [junit4]   2> 32923 INFO  (parallelCoreAdminExecutor-22-thread-1-processing-n:127.0.0.1:59601_solr 195bbadd-9c5c-4462-a990-d44a2677490420970050602881683 CREATE) [n:127.0.0.1:59601_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 32906 INFO  (parallelCoreAdminExecutor-27-thread-1-processing-n:127.0.0.1:42367_solr 195bbadd-9c5c-4462-a990-d44a2677490420970050616037112 CREATE) [n:127.0.0.1:42367_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=195bbadd-9c5c-4462-a990-d44a2677490420970050616037112&coreNodeName=core_node5&name=testcollection_shard1_replica_n3&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 32924 INFO  (parallelCoreAdminExecutor-27-thread-1-processing-n:127.0.0.1:42367_solr 195bbadd-9c5c-4462-a990-d44a2677490420970050616037112 CREATE) [n:127.0.0.1:42367_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 32958 INFO  (qtp2006433491-118) [n:127.0.0.1:42367_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=195bbadd-9c5c-4462-a990-d44a2677490420970050630310431&coreNodeName=core_node8&name=testcollection_shard2_replica_n6&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=0 QTime=49
   [junit4]   2> 32970 INFO  (parallelCoreAdminExecutor-27-thread-2-processing-n:127.0.0.1:42367_solr 195bbadd-9c5c-4462-a990-d44a2677490420970050630310431 CREATE) [n:127.0.0.1:42367_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=195bbadd-9c5c-4462-a990-d44a2677490420970050630310431&coreNodeName=core_node8&name=testcollection_shard2_replica_n6&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 32981 INFO  (qtp1705980688-112) [n:127.0.0.1:59601_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=195bbadd-9c5c-4462-a990-d44a2677490420970050622768873&coreNodeName=core_node7&name=testcollection_shard2_replica_n4&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=0 QTime=50
   [junit4]   2> 33021 INFO  (parallelCoreAdminExecutor-22-thread-2-processing-n:127.0.0.1:59601_solr 195bbadd-9c5c-4462-a990-d44a2677490420970050622768873 CREATE) [n:127.0.0.1:59601_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=195bbadd-9c5c-4462-a990-d44a2677490420970050622768873&coreNodeName=core_node7&name=testcollection_shard2_replica_n4&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 33028 INFO  (qtp1705980688-109) [n:127.0.0.1:59601_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for :

[...truncated too long message...]

ding=US-ASCII
   [junit4] ERROR   0.00s J0 | TestLargeCluster (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster: 
   [junit4]    >    1) Thread[id=24944, name=AutoscalingActionExecutor-8716-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]
   [junit4]    >         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:131)
   [junit4]    >         at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:110)
   [junit4]    >         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)
   [junit4]    >         at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:108)
   [junit4]    >         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)
   [junit4]    >         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:74)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Row.copy(Row.java:91)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.lambda$getMatrixCopy$1(Policy.java:297)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session$$Lambda$584/1971910727.apply(Unknown Source)
   [junit4]    >         at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
   [junit4]    >         at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
   [junit4]    >         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
   [junit4]    >         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
   [junit4]    >         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
   [junit4]    >         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
   [junit4]    >         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:298)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:287)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Row.addReplica(Row.java:122)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:59)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$397/793967629.run(Unknown Source)
   [junit4]    >         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]    >         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$119/1716245607.run(Unknown Source)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([EC7BE4E50471C09A]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=24944, name=AutoscalingActionExecutor-8716-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]
   [junit4]    >         at java.util.ArrayList.iterator(ArrayList.java:834)
   [junit4]    >         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:131)
   [junit4]    >         at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:110)
   [junit4]    >         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)
   [junit4]    >         at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:108)
   [junit4]    >         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)
   [junit4]    >         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:74)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Row.copy(Row.java:91)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.lambda$getMatrixCopy$1(Policy.java:297)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session$$Lambda$584/1971910727.apply(Unknown Source)
   [junit4]    >         at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
   [junit4]    >         at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
   [junit4]    >         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
   [junit4]    >         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
   [junit4]    >         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
   [junit4]    >         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
   [junit4]    >         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:298)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:287)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Row.removeReplica(Row.java:156)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:60)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$397/793967629.run(Unknown Source)
   [junit4]    >         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]    >         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$119/1716245607.run(Unknown Source)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([EC7BE4E50471C09A]:0)
   [junit4] Completed [635/784 (8!)] on J0 in 124.93s, 4 tests, 2 errors <<< FAILURES!

[...truncated 7854 lines...]
   [junit4] Suite: org.apache.solr.ltr.TestLTRReRankingPipeline
   [junit4]   2> 1    INFO  (SUITE-TestLTRReRankingPipeline-seed#[75AA0CB61F054D50]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 2034 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[75AA0CB61F054D50]) [    ] o.a.s.l.TestLTRReRankingPipeline rerank 1 documents 
   [junit4]   2> 2036 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[75AA0CB61F054D50]) [    ] o.a.s.l.TestLTRReRankingPipeline doc 0 in pos 0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTRReRankingPipeline -Dtests.method=testDifferentTopN -Dtests.seed=75AA0CB61F054D50 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=pl -Dtests.timezone=Atlantic/South_Georgia -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.15s J0 | TestLTRReRankingPipeline.testDifferentTopN <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1.0> but was:<0.0>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([75AA0CB61F054D50:840B7EE62ABE87C2]:0)
   [junit4]    > 	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {field=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), id=PostingsFormat(name=Memory), final-score=PostingsFormat(name=Memory)}, docValues:{final-score=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=291, maxMBSortInHeap=7.736832951849775, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2f72910b), locale=pl, timezone=Atlantic/South_Georgia
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=236707856,total=253231104
   [junit4]   2> NOTE: All tests run in this JVM: [TestLTRReRankingPipeline]
   [junit4] Completed [2/37 (1!)] on J0 in 2.76s, 3 tests, 1 failure <<< FAILURES!

[...truncated 42044 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 9 - Failure

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/9/

8 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup

Error Message:
cleanup action didn't run

Stack Trace:
java.lang.AssertionError: cleanup action didn't run
	at __randomizedtesting.SeedInfo.seed([A86D2A7D5E1CA120:B541EA0F3F5F862B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup(ScheduledMaintenanceTriggerTest.java:197)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue

Error Message:
action wasn't interrupted

Stack Trace:
java.lang.AssertionError: action wasn't interrupted
	at __randomizedtesting.SeedInfo.seed([A86D2A7D5E1CA120:61D868D3577B67D5]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue(TriggerIntegrationTest.java:752)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestLargeCluster

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster:     1) Thread[id=24465, name=AutoscalingActionExecutor-8666-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]         at java.util.ArrayList.iterator(ArrayList.java:834)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:131)         at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:110)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)         at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:108)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:74)         at org.apache.solr.client.solrj.cloud.autoscaling.Row.copy(Row.java:91)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.lambda$getMatrixCopy$1(Policy.java:297)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session$$Lambda$453/584734946.apply(Unknown Source)         at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)         at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:298)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:287)         at org.apache.solr.client.solrj.cloud.autoscaling.Row.removeReplica(Row.java:156)         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:60)         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)         at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)         at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$508/1781589047.run(Unknown Source)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$119/612221234.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster: 
   1) Thread[id=24465, name=AutoscalingActionExecutor-8666-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]
        at java.util.ArrayList.iterator(ArrayList.java:834)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:131)
        at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:110)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)
        at org.apache.solr.common.util.Utils.makeDeepCopy(Utils.java:108)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:92)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:74)
        at org.apache.solr.client.solrj.cloud.autoscaling.Row.copy(Row.java:91)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.lambda$getMatrixCopy$1(Policy.java:297)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session$$Lambda$453/584734946.apply(Unknown Source)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:298)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:287)
        at org.apache.solr.client.solrj.cloud.autoscaling.Row.removeReplica(Row.java:156)
        at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:60)
        at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)
        at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)
        at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$508/1781589047.run(Unknown Source)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$119/612221234.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([A86D2A7D5E1CA120]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestLargeCluster

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=24465, name=AutoscalingActionExecutor-8666-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:90)         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:74)         at org.apache.solr.client.solrj.cloud.autoscaling.Row.copy(Row.java:91)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.lambda$getMatrixCopy$1(Policy.java:297)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session$$Lambda$453/584734946.apply(Unknown Source)         at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)         at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:298)         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:287)         at org.apache.solr.client.solrj.cloud.autoscaling.Row.removeReplica(Row.java:156)         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:60)         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)         at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)         at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$508/1781589047.run(Unknown Source)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$119/612221234.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=24465, name=AutoscalingActionExecutor-8666-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:90)
        at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:74)
        at org.apache.solr.client.solrj.cloud.autoscaling.Row.copy(Row.java:91)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.lambda$getMatrixCopy$1(Policy.java:297)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session$$Lambda$453/584734946.apply(Unknown Source)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:298)
        at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:287)
        at org.apache.solr.client.solrj.cloud.autoscaling.Row.removeReplica(Row.java:156)
        at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:60)
        at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)
        at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)
        at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$508/1781589047.run(Unknown Source)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$119/612221234.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([A86D2A7D5E1CA120]:0)


FAILED:  org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([A86D2A7D5E1CA120:845F5DF0D2D8D7A4]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:904)
	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

	request was:q=cat:jrzf12cwnsz1hcj5rg9p+u8c6eblbjvf1bs8ysglp+q37fg815srgv9o3wbbn5+2cdayz7makuvvwtj6fx3+x10v4y8rus28ragiym7p&wt=xml
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:897)
	... 40 more


FAILED:  org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN

Error Message:
expected:<1.0> but was:<0.0>

Stack Trace:
java.lang.AssertionError: expected:<1.0> but was:<0.0>
	at __randomizedtesting.SeedInfo.seed([726283AAA62ED3B7:83C3F1FA93951925]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:443)
	at org.junit.Assert.assertEquals(Assert.java:512)
	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload

Error Message:
Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>

Stack Trace:
java.lang.AssertionError: Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>
	at __randomizedtesting.SeedInfo.seed([A86D2A7D5E1CA120:AAD4F1112D5F7DED]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload(TestJmxIntegration.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart

Error Message:
IOException occured when talking to server at: http://127.0.0.1:45222/solr/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:45222/solr/collection1
	at __randomizedtesting.SeedInfo.seed([A86D2A7D5E1CA120:709AEE99F5C7637C]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
	at org.apache.solr.handler.TestReplicationHandler.index(TestReplicationHandler.java:180)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart(TestReplicationHandler.java:643)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:45222 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 47 more




Build Log:
[...truncated 12499 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_A86D2A7D5E1CA120-001/init-core-data-001
   [junit4]   2> 346450 WARN  (SUITE-TriggerIntegrationTest-seed#[A86D2A7D5E1CA120]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
   [junit4]   2> 346451 INFO  (SUITE-TriggerIntegrationTest-seed#[A86D2A7D5E1CA120]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 346453 INFO  (SUITE-TriggerIntegrationTest-seed#[A86D2A7D5E1CA120]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 346453 INFO  (SUITE-TriggerIntegrationTest-seed#[A86D2A7D5E1CA120]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 346454 INFO  (SUITE-TriggerIntegrationTest-seed#[A86D2A7D5E1CA120]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_A86D2A7D5E1CA120-001/tempDir-001
   [junit4]   2> 346454 INFO  (SUITE-TriggerIntegrationTest-seed#[A86D2A7D5E1CA120]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 346455 INFO  (Thread-458) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 346455 INFO  (Thread-458) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 346460 ERROR (Thread-458) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 346555 INFO  (SUITE-TriggerIntegrationTest-seed#[A86D2A7D5E1CA120]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:52337
   [junit4]   2> 346558 INFO  (zkConnectionManagerCallback-559-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 346571 INFO  (jetty-launcher-556-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T08:27:37+11:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 346574 INFO  (jetty-launcher-556-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T08:27:37+11:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 346575 INFO  (jetty-launcher-556-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 346575 INFO  (jetty-launcher-556-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 346575 INFO  (jetty-launcher-556-thread-1) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 346575 INFO  (jetty-launcher-556-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@15377e22{/solr,null,AVAILABLE}
   [junit4]   2> 346580 INFO  (jetty-launcher-556-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3418cfbe{SSL,[ssl, http/1.1]}{127.0.0.1:60093}
   [junit4]   2> 346581 INFO  (jetty-launcher-556-thread-1) [    ] o.e.j.s.Server Started @349793ms
   [junit4]   2> 346581 INFO  (jetty-launcher-556-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60093}
   [junit4]   2> 346581 ERROR (jetty-launcher-556-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 346581 INFO  (jetty-launcher-556-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 346581 INFO  (jetty-launcher-556-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 346581 INFO  (jetty-launcher-556-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 346581 INFO  (jetty-launcher-556-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-09T23:20:51.848Z
   [junit4]   2> 346584 INFO  (jetty-launcher-556-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 346584 INFO  (jetty-launcher-556-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 346584 INFO  (jetty-launcher-556-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 346584 INFO  (jetty-launcher-556-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@72fdb681{/solr,null,AVAILABLE}
   [junit4]   2> 346585 INFO  (jetty-launcher-556-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6d63c3ca{SSL,[ssl, http/1.1]}{127.0.0.1:47907}
   [junit4]   2> 346585 INFO  (jetty-launcher-556-thread-2) [    ] o.e.j.s.Server Started @349797ms
   [junit4]   2> 346585 INFO  (jetty-launcher-556-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=47907}
   [junit4]   2> 346586 ERROR (jetty-launcher-556-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 346586 INFO  (jetty-launcher-556-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 346586 INFO  (jetty-launcher-556-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 346586 INFO  (jetty-launcher-556-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 346586 INFO  (jetty-launcher-556-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-09T23:20:51.853Z
   [junit4]   2> 346591 INFO  (zkConnectionManagerCallback-563-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 346592 INFO  (jetty-launcher-556-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 346593 INFO  (zkConnectionManagerCallback-561-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 346594 INFO  (jetty-launcher-556-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 346603 INFO  (jetty-launcher-556-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52337/solr
   [junit4]   2> 346604 INFO  (jetty-launcher-556-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52337/solr
   [junit4]   2> 346608 INFO  (zkConnectionManagerCallback-568-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 346617 INFO  (zkConnectionManagerCallback-571-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 346622 INFO  (zkConnectionManagerCallback-573-thread-1-processing-n:127.0.0.1:47907_solr) [n:127.0.0.1:47907_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 346622 INFO  (zkConnectionManagerCallback-575-thread-1-processing-n:127.0.0.1:60093_solr) [n:127.0.0.1:60093_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 346809 INFO  (jetty-launcher-556-thread-2) [n:127.0.0.1:47907_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 346810 INFO  (jetty-launcher-556-thread-2) [n:127.0.0.1:47907_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:47907_solr
   [junit4]   2> 346811 INFO  (jetty-launcher-556-thread-2) [n:127.0.0.1:47907_solr    ] o.a.s.c.Overseer Overseer (id=73426611869843461-127.0.0.1:47907_solr-n_0000000000) starting
   [junit4]   2> 346818 INFO  (jetty-launcher-556-thread-1) [n:127.0.0.1:60093_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 346819 INFO  (jetty-launcher-556-thread-1) [n:127.0.0.1:60093_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60093_solr
   [junit4]   2> 346835 INFO  (zkCallback-574-thread-1-processing-n:127.0.0.1:60093_solr) [n:127.0.0.1:60093_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 346839 INFO  (jetty-launcher-556-thread-2) [n:127.0.0.1:47907_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47907_solr
   [junit4]   2> 346839 INFO  (zkCallback-572-thread-1-processing-n:127.0.0.1:47907_solr) [n:127.0.0.1:47907_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 346840 INFO  (zkCallback-574-thread-1-processing-n:127.0.0.1:60093_solr) [n:127.0.0.1:60093_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 346841 DEBUG (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 346843 DEBUG (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 346844 DEBUG (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:60093_solr]
   [junit4]   2> 346844 DEBUG (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 346844 DEBUG (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 346851 DEBUG (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 346851 DEBUG (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 346852 DEBUG (ScheduledTrigger-832-thread-2) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 346853 INFO  (OverseerStateUpdate-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 346907 INFO  (jetty-launcher-556-thread-1) [n:127.0.0.1:60093_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60093.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 346918 INFO  (jetty-launcher-556-thread-1) [n:127.0.0.1:60093_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60093.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 346918 INFO  (jetty-launcher-556-thread-1) [n:127.0.0.1:60093_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60093.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 346920 INFO  (jetty-launcher-556-thread-1) [n:127.0.0.1:60093_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_A86D2A7D5E1CA120-001/tempDir-001/node1/.
   [junit4]   2> 346958 INFO  (jetty-launcher-556-thread-2) [n:127.0.0.1:47907_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_47907.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 346970 INFO  (jetty-launcher-556-thread-2) [n:127.0.0.1:47907_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_47907.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 346970 INFO  (jetty-launcher-556-thread-2) [n:127.0.0.1:47907_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_47907.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 346972 INFO  (jetty-launcher-556-thread-2) [n:127.0.0.1:47907_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_A86D2A7D5E1CA120-001/tempDir-001/node2/.
   [junit4]   2> 347001 INFO  (zkConnectionManagerCallback-581-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 347006 INFO  (zkConnectionManagerCallback-585-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 347008 INFO  (SUITE-TriggerIntegrationTest-seed#[A86D2A7D5E1CA120]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 347010 INFO  (SUITE-TriggerIntegrationTest-seed#[A86D2A7D5E1CA120]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:52337/solr ready
   [junit4]   2> 347055 INFO  (qtp457769581-2195) [n:127.0.0.1:47907_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 347077 INFO  (qtp483143024-2186) [n:127.0.0.1:60093_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 347077 DEBUG (qtp457769581-2194) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 347078 INFO  (qtp457769581-2194) [n:127.0.0.1:47907_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=38
   [junit4]   2> 347078 DEBUG (zkCallback-572-thread-1-processing-n:127.0.0.1:47907_solr) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 347080 DEBUG (zkCallback-572-thread-1-processing-n:127.0.0.1:47907_solr) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:60093_solr, 127.0.0.1:47907_solr]
   [junit4]   2> 347080 DEBUG (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 347088 DEBUG (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 347089 DEBUG (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 347104 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSearchRate
   [junit4]   2> 347105 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.a.TriggerIntegrationTest Found 2 jetty instances running
   [junit4]   2> 347106 INFO  (zkConnectionManagerCallback-587-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 347108 INFO  (qtp457769581-2192) [n:127.0.0.1:47907_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 347117 INFO  (qtp457769581-2192) [n:127.0.0.1:47907_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=9
   [junit4]   2> 347119 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.a.ScheduledTriggers -- removing trigger: .auto_add_replicas
   [junit4]   2> 347122 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 3
   [junit4]   2> 347122 DEBUG (zkCallback-572-thread-1-processing-n:127.0.0.1:47907_solr) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 347122 DEBUG (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 347122 DEBUG (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 347122 DEBUG (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 347123 DEBUG (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 347123 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 347125 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@6d63c3ca{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 347126 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=657547215
   [junit4]   2> 347126 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 347127 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@45ca6c43: rootName = solr_47907, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@44e9451c
   [junit4]   2> 347135 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 347135 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@76ccca87: rootName = solr_47907, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@5171e23c
   [junit4]   2> 347141 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 347141 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6043fe42: rootName = solr_47907, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@554e7cca
   [junit4]   2> 347145 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:47907_solr
   [junit4]   2> 347147 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 347147 INFO  (zkCallback-572-thread-1-processing-n:127.0.0.1:47907_solr) [n:127.0.0.1:47907_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 347147 INFO  (zkCallback-574-thread-1-processing-n:127.0.0.1:60093_solr) [n:127.0.0.1:60093_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 347148 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 347148 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.Overseer Overseer (id=73426611869843461-127.0.0.1:47907_solr-n_0000000000) closing
   [junit4]   2> 347148 INFO  (OverseerStateUpdate-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:47907_solr
   [junit4]   2> 347149 INFO  (zkCallback-584-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 347148 INFO  (zkCallback-584-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 347152 WARN  (OverseerAutoScalingTriggerThread-73426611869843461-127.0.0.1:47907_solr-n_0000000000) [n:127.0.0.1:47907_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 347152 DEBUG (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 347153 INFO  (zkCallback-574-thread-1-processing-n:127.0.0.1:60093_solr) [n:127.0.0.1:60093_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60093_solr
   [junit4]   2> 347154 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@72fdb681{/solr,null,UNAVAILABLE}
   [junit4]   2> 347155 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 347155 INFO  (zkCallback-574-thread-1-processing-n:127.0.0.1:60093_solr) [n:127.0.0.1:60093_solr    ] o.a.s.c.Overseer Overseer (id=73426611869843462-127.0.0.1:60093_solr-n_0000000001) starting
   [junit4]   2> 347167 DEBUG (OverseerAutoScalingTriggerThread-73426611869843462-127.0.0.1:60093_solr-n_0000000001) [n:127.0.0.1:60093_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 347170 DEBUG (OverseerAutoScalingTriggerThread-73426611869843462-127.0.0.1:60093_solr-n_0000000001) [n:127.0.0.1:60093_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 347171 DEBUG (OverseerAutoScalingTriggerThread-73426611869843462-127.0.0.1:60093_solr-n_0000000001) [n:127.0.0.1:60093_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:60093_solr]
   [junit4]   2> 347171 DEBUG (OverseerAutoScalingTriggerThread-73426611869843462-127.0.0.1:60093_solr-n_0000000001) [n:127.0.0.1:60093_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion -1
   [junit4]   2> 347171 DEBUG (OverseerAutoScalingTriggerThread-73426611869843462-127.0.0.1:60093_solr-n_0000000001) [n:127.0.0.1:60093_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 347175 DEBUG (OverseerAutoScalingTriggerThread-73426611869843462-127.0.0.1:60093_solr-n_0000000001) [n:127.0.0.1:60093_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 347175 DEBUG (OverseerAutoScalingTriggerThread-73426611869843462-127.0.0.1:60093_solr-n_0000000001) [n:127.0.0.1:60093_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 347175 DEBUG (ScheduledTrigger-841-thread-1) [n:127.0.0.1:60093_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 348183 DEBUG (ScheduledTrigger-841-thread-3) [n:127.0.0.1:60093_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 349183 DEBUG (ScheduledTrigger-841-thread-4) [n:127.0.0.1:60093_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 350184 DEBUG (ScheduledTrigger-841-thread-3) [n:127.0.0.1:60093_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 351184 DEBUG (ScheduledTrigger-841-thread-4) [n:127.0.0.1:60093_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 352159 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T08:27:37+11:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 352162 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 352162 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 352162 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 352163 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@239dfa03{/solr,null,AVAILABLE}
   [junit4]   2> 352164 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7cdc3e6{SSL,[ssl, http/1.1]}{127.0.0.1:46613}
   [junit4]   2> 352164 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.Server Started @355376ms
   [junit4]   2> 352164 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46613}
   [junit4]   2> 352165 ERROR (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 352165 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 352165 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 352165 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 352166 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-09T23:20:57.433Z
   [junit4]   2> 352168 INFO  (zkConnectionManagerCallback-590-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 352169 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 352181 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52337/solr
   [junit4]   2> 352183 INFO  (zkConnectionManagerCallback-594-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 352184 DEBUG (ScheduledTrigger-841-thread-3) [n:127.0.0.1:60093_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 352187 INFO  (zkConnectionManagerCallback-596-thread-1-processing-n:127.0.0.1:46613_solr) [n:127.0.0.1:46613_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 352200 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:46613_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 352203 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:46613_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 352206 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:46613_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46613_solr
   [junit4]   2> 352208 INFO  (zkCallback-574-thread-1-processing-n:127.0.0.1:60093_solr) [n:127.0.0.1:60093_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 352208 INFO  (zkCallback-584-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 352208 INFO  (zkCallback-584-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 352212 INFO  (zkCallback-595-thread-1-processing-n:127.0.0.1:46613_solr) [n:127.0.0.1:46613_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 352291 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:46613_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46613.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 352306 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:46613_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46613.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 352306 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:46613_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46613.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 352309 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:46613_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_A86D2A7D5E1CA120-001/tempDir-001/node3/.
   [junit4]   2> 352365 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T08:27:37+11:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 352366 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 352366 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 352366 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 352367 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2ab54ed2{/solr,null,AVAILABLE}
   [junit4]   2> 352368 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@79fe08b9{SSL,[ssl, http/1.1]}{127.0.0.1:53439}
   [junit4]   2> 352368 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.Server Started @355580ms
   [junit4]   2> 352368 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53439}
   [junit4]   2> 352369 ERROR (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 352369 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 352369 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 352369 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 352369 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-09T23:20:57.636Z
   [junit4]   2> 352371 INFO  (zkConnectionManagerCallback-600-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 352372 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 352383 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52337/solr
   [junit4]   2> 352385 INFO  (zkConnectionManagerCallback-604-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 352388 INFO  (zkConnectionManagerCallback-606-thread-1-processing-n:127.0.0.1:53439_solr) [n:127.0.0.1:53439_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 352398 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:53439_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 352401 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:53439_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 352404 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:53439_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53439_solr
   [junit4]   2> 352406 INFO  (zkCallback-574-thread-1-processing-n:127.0.0.1:60093_solr) [n:127.0.0.1:60093_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 352406 INFO  (zkCallback-595-thread-1-processing-n:127.0.0.1:46613_solr) [n:127.0.0.1:46613_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 352408 INFO  (zkCallback-584-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 352408 INFO  (zkCallback-584-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 352409 INFO  (zkCallback-605-thread-1-processing-n:127.0.0.1:53439_solr) [n:127.0.0.1:53439_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 352520 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:53439_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53439.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 352535 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:53439_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53439.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 352536 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:53439_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53439.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 352538 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:53439_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_A86D2A7D5E1CA120-001/tempDir-001/node4/.
   [junit4]   2> 352610 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T08:27:37+11:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 352612 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 352612 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 352612 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 352613 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1f4f3c7e{/solr,null,AVAILABLE}
   [junit4]   2> 352614 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7908a5dd{SSL,[ssl, http/1.1]}{127.0.0.1:33714}
   [junit4]   2> 352614 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.Server Started @355826ms
   [junit4]   2> 352614 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33714}
   [junit4]   2> 352614 ERROR (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 352614 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 352614 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 352614 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 352615 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-09T23:20:57.882Z
   [junit4]   2> 352617 INFO  (zkConnectionManagerCallback-610-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 352618 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 352629 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52337/solr
   [junit4]   2> 352631 INFO  (zkConnectionManagerCallback-614-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 352636 INFO  (zkConnectionManagerCallback-616-thread-1-processing-n:127.0.0.1:33714_solr) [n:127.0.0.1:33714_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 352644 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:33714_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 352648 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:33714_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 352651 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:33714_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33714_solr
   [junit4]   2> 352653 INFO  (zkCallback-574-thread-1-processing-n:127.0.0.1:60093_solr) [n:127.0.0.1:60093_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 352653 INFO  (zkCallback-605-thread-1-processing-n:127.0.0.1:53439_solr) [n:127.0.0.1:53439_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 352653 INFO  (zkCallback-595-thread-1-processing-n:127.0.0.1:46613_solr) [n:127.0.0.1:46613_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 352653 INFO  (zkCallback-584-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 352653 INFO  (zkCallback-584-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 352659 INFO  (zkCallback-615-thread-1-processing-n:127.0.0.1:33714_solr) [n:127.0.0.1:33714_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 352749 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:33714_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33714.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 352764 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:33714_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33714.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 352765 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:33714_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33714.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 352767 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:33714_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_A86D2A7D5E1CA120-001/tempDir-001/node5/.
   [junit4]   2> 352857 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T08:27:37+11:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 352858 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 352858 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 352858 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 352859 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2d7d5bac{/solr,null,AVAILABLE}
   [junit4]   2> 352860 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@30075c67{SSL,[ssl, http/1.1]}{127.0.0.1:44082}
   [junit4]   2> 352860 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.e.j.s.Server Started @356072ms
   [junit4]   2> 352860 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44082}
   [junit4]   2> 352860 ERROR (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 352860 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 352860 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 352860 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 352860 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-09T23:20:58.127Z
   [junit4]   2> 352863 INFO  (zkConnectionManagerCallback-620-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 352864 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 352875 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52337/solr
   [junit4]   2> 352877 INFO  (zkConnectionManagerCallback-624-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 352882 INFO  (zkConnectionManagerCallback-626-thread-1-processing-n:127.0.0.1:44082_solr) [n:127.0.0.1:44082_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 352891 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:44082_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 352894 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:44082_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 352897 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:44082_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44082_solr
   [junit4]   2> 352899 INFO  (zkCallback-595-thread-1-processing-n:127.0.0.1:46613_solr) [n:127.0.0.1:46613_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 352902 INFO  (zkCallback-605-thread-1-processing-n:127.0.0.1:53439_solr) [n:127.0.0.1:53439_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 352902 INFO  (zkCallback-584-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 352902 INFO  (zkCallback-584-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 352902 INFO  (zkCallback-615-thread-1-processing-n:127.0.0.1:33714_solr) [n:127.0.0.1:33714_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 352903 INFO  (zkCallback-574-thread-1-processing-n:127.0.0.1:60093_solr) [n:127.0.0.1:60093_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 352904 INFO  (zkCallback-625-thread-1-processing-n:127.0.0.1:44082_solr) [n:127.0.0.1:44082_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 353034 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:44082_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44082.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 353049 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:44082_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44082.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 353049 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:44082_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44082.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65514c15
   [junit4]   2> 353051 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[A86D2A7D5E1CA120]) [n:127.0.0.1:44082_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_A86D2A7D5E1

[...truncated too long message...]

java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
   [junit4]    >         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
   [junit4]    >         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:298)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:287)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Row.removeReplica(Row.java:156)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:60)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$508/1781589047.run(Unknown Source)
   [junit4]    >         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]    >         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$119/612221234.run(Unknown Source)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A86D2A7D5E1CA120]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=24465, name=AutoscalingActionExecutor-8666-thread-1, state=RUNNABLE, group=TGRP-TestLargeCluster]
   [junit4]    >         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:90)
   [junit4]    >         at org.apache.solr.common.util.Utils.getDeepCopy(Utils.java:74)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Row.copy(Row.java:91)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.lambda$getMatrixCopy$1(Policy.java:297)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session$$Lambda$453/584734946.apply(Unknown Source)
   [junit4]    >         at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
   [junit4]    >         at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
   [junit4]    >         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
   [junit4]    >         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
   [junit4]    >         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
   [junit4]    >         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
   [junit4]    >         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.getMatrixCopy(Policy.java:298)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.copy(Policy.java:287)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Row.removeReplica(Row.java:156)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.tryEachNode(MoveReplicaSuggester.java:60)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.MoveReplicaSuggester.init(MoveReplicaSuggester.java:34)
   [junit4]    >         at org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:129)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:98)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:307)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$508/1781589047.run(Unknown Source)
   [junit4]    >         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]    >         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$119/612221234.run(Unknown Source)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A86D2A7D5E1CA120]:0)
   [junit4] Completed [600/784 (6!)] on J2 in 124.12s, 4 tests, 2 errors <<< FAILURES!

[...truncated 8004 lines...]
   [junit4] Suite: org.apache.solr.ltr.TestLTRReRankingPipeline
   [junit4]   2> 13968 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[726283AAA62ED3B7]) [    ] o.a.s.l.TestLTRReRankingPipeline rerank 1 documents 
   [junit4]   2> 13974 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[726283AAA62ED3B7]) [    ] o.a.s.l.TestLTRReRankingPipeline doc 0 in pos 0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTRReRankingPipeline -Dtests.method=testDifferentTopN -Dtests.seed=726283AAA62ED3B7 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=es-MX -Dtests.timezone=America/Mexico_City -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.28s J0 | TestLTRReRankingPipeline.testDifferentTopN <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1.0> but was:<0.0>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([726283AAA62ED3B7:83C3F1FA93951925]:0)
   [junit4]    > 	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@3c8eac5d), locale=es-MX, timezone=America/Mexico_City
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=244342200,total=335020032
   [junit4]   2> NOTE: All tests run in this JVM: [TestDefaultWrapperModel, TestRankingFeature, TestLTRReRankingPipeline]
   [junit4] Completed [8/37 (1!)] on J0 in 0.43s, 3 tests, 1 failure <<< FAILURES!

[...truncated 38481 lines...]
-ecj-javadoc-lint-tests:
    [mkdir] Created dir: /tmp/ecj2137110498
 [ecj-lint] Compiling 877 source files to /tmp/ecj2137110498
 [ecj-lint] invalid Class-Path header in manifest of jar file: /x1/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
 [ecj-lint] invalid Class-Path header in manifest of jar file: /x1/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
 [ecj-lint] ----------
 [ecj-lint] 1. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test/org/apache/solr/analysis/TokenizerChainTest.java (at line 37)
 [ecj-lint] 	TokenizerChain tokenizerChain = new TokenizerChain(
 [ecj-lint] 	               ^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'tokenizerChain' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 2. ERROR in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test/org/apache/solr/cloud/TestCloudConsistency.java (at line 23)
 [ecj-lint] 	import java.net.URL;
 [ecj-lint] 	       ^^^^^^^^^^^^
 [ecj-lint] The import java.net.URL is never used
 [ecj-lint] ----------
 [ecj-lint] 3. ERROR in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test/org/apache/solr/cloud/TestCloudConsistency.java (at line 39)
 [ecj-lint] 	import org.apache.solr.common.cloud.ZkStateReader;
 [ecj-lint] 	       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import org.apache.solr.common.cloud.ZkStateReader is never used
 [ecj-lint] ----------
 [ecj-lint] 3 problems (2 errors, 1 warning)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/build.xml:627: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/build.xml:101: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build.xml:682: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/common-build.xml:2088: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/common-build.xml:2121: Compile failed; see the compiler error output for details.

Total time: 103 minutes 20 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 8 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/8/

5 tests failed.
FAILED:  org.apache.solr.cloud.TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader

Error Message:
Doc with id=4 not found in https://127.0.0.1:51053/solr/outOfSyncReplicasCannotBecomeLeader-false due to: Path not found: /id; rsp={doc=null}

Stack Trace:
java.lang.AssertionError: Doc with id=4 not found in https://127.0.0.1:51053/solr/outOfSyncReplicasCannotBecomeLeader-false due to: Path not found: /id; rsp={doc=null}
	at __randomizedtesting.SeedInfo.seed([9269A545F6156241:EC82855535726D7B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.TestCloudConsistency.assertDocExists(TestCloudConsistency.java:250)
	at org.apache.solr.cloud.TestCloudConsistency.assertDocsExistInAllReplicas(TestCloudConsistency.java:234)
	at org.apache.solr.cloud.TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader(TestCloudConsistency.java:127)
	at org.apache.solr.cloud.TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader(TestCloudConsistency.java:90)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([9269A545F6156241:BE5BD2C87AD114C5]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:904)
	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:260)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">15</int></lst><result name="response" numFound="0" start="0"></result>
</response>

	request was:q=int_i:27&wt=xml
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:897)
	... 40 more


FAILED:  org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN

Error Message:
expected:<1.0> but was:<0.0>

Stack Trace:
java.lang.AssertionError: expected:<1.0> but was:<0.0>
	at __randomizedtesting.SeedInfo.seed([369054BF4F2D0802:C73126EF7A96C290]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:443)
	at org.junit.Assert.assertEquals(Assert.java:512)
	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload

Error Message:
Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>

Stack Trace:
java.lang.AssertionError: Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>
	at __randomizedtesting.SeedInfo.seed([9269A545F6156241:90D07E298556BE8C]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload(TestJmxIntegration.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart

Error Message:
IOException occured when talking to server at: http://127.0.0.1:59132/solr/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:59132/solr/collection1
	at __randomizedtesting.SeedInfo.seed([9269A545F6156241:4A9E61A15DCEA01D]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
	at org.apache.solr.handler.TestReplicationHandler.index(TestReplicationHandler.java:180)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart(TestReplicationHandler.java:643)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:59132 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 47 more




Build Log:
[...truncated 12073 lines...]
   [junit4] Suite: org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_9269A545F6156241-001/init-core-data-001
   [junit4]   2> 19417 WARN  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
   [junit4]   2> 19417 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 19419 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 19419 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 19420 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 19420 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 19467 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 19547 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 19743 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 19898 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5076aee3
   [junit4]   2> 19920 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5076aee3
   [junit4]   2> 19920 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5076aee3
   [junit4]   2> 19924 INFO  (coreLoadExecutor-84-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 19925 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 19962 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 19990 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 20122 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 20157 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 20157 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5076aee3
   [junit4]   2> 20158 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 20158 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_9269A545F6156241-001/init-core-data-001/]
   [junit4]   2> 20162 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=24, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3171226400284961]
   [junit4]   2> 20282 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 20282 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 20290 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 20290 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 20292 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=25, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7373187994007929]
   [junit4]   2> 20293 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@38e2c5ae[collection1] main]
   [junit4]   2> 20295 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 20314 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 20365 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 20365 INFO  (searcherExecutor-85-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 20366 INFO  (coreLoadExecutor-84-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1594358524124594176
   [junit4]   2> 20366 INFO  (searcherExecutor-85-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 20366 INFO  (searcherExecutor-85-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 20366 INFO  (searcherExecutor-85-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 20366 INFO  (searcherExecutor-85-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 20367 INFO  (searcherExecutor-85-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 20367 INFO  (searcherExecutor-85-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 20368 INFO  (searcherExecutor-85-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 20368 INFO  (searcherExecutor-85-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 20370 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 20373 INFO  (searcherExecutor-85-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@38e2c5ae[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 20387 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testWrongAtomicOpPassed-seed#[9269A545F6156241]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testWrongAtomicOpPassed
   [junit4]   2> 20389 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testWrongAtomicOpPassed-seed#[9269A545F6156241]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testWrongAtomicOpPassed
   [junit4]   2> 20393 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMultipleThreads
   [junit4]   2> 20407 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 20409 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@49e60e2c[collection1] realtime]
   [junit4]   2> 20409 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 15
   [junit4]   2> 20530 WARN  (Thread-24) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1594358524190654464 actual=1594358524290269184Retrying with new version .... (1)
   [junit4]   2> 20532 WARN  (Thread-25) [    ] o.a.s.u.p.AtomicUpdateProcessorFactory Atomic update failed due to version conflict for 10 expected=1594358524285026305 actual=1594358524290269184Retrying with new version .... (1)
   [junit4]   2> 20533 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1594358524299706368,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 20533 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@5cf2ad02 commitCommandVersion:1594358524299706368
   [junit4]   2> 20662 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@23842297[collection1] main]
   [junit4]   2> 20663 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 20668 INFO  (searcherExecutor-85-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@23842297[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C10/9:delGen=1)))}
   [junit4]   2> 20669 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 135
   [junit4]   2> 20695 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:10&wt=xml} hits=1 status=0 QTime=22
   [junit4]   2> 20789 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:jk5c6h9hgculb9wj9uy4+ksvngr7zj5aqn0jt5n1u+buuieq6rkqpygipb9vkq+qjs567r6tazgrkd0opvv+vwjkx7wi5vd3a97p196q&wt=xml} hits=1 status=0 QTime=60
   [junit4]   2> 20813 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=int_i:27&wt=xml} hits=0 status=0 QTime=15
   [junit4]   2> 20814 ERROR (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.SolrTestCaseJ4 REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]   2> 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">15</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2> 
   [junit4]   2> 	request was:q=int_i:27&wt=xml
   [junit4]   2> 20815 ERROR (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.SolrTestCaseJ4 REQUEST FAILED: q=int_i:27&wt=xml:java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]   2> 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">15</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2> 
   [junit4]   2> 	request was:q=int_i:27&wt=xml
   [junit4]   2> 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:897)
   [junit4]   2> 	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:260)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 20815 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testMultipleThreads-seed#[9269A545F6156241]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testMultipleThreads
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=AtomicUpdateProcessorFactoryTest -Dtests.method=testMultipleThreads -Dtests.seed=9269A545F6156241 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=nl-BE -Dtests.timezone=Africa/Nairobi -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.43s J2 | AtomicUpdateProcessorFactoryTest.testMultipleThreads <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9269A545F6156241:BE5BD2C87AD114C5]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:904)
   [junit4]    > 	at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:260)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]    > 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]    > <response>
   [junit4]    > <lst name="responseHeader"><int name="status">0</int><int name="QTime">15</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]    > </response>
   [junit4]    > 	request was:q=int_i:27&wt=xml
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:897)
   [junit4]    > 	... 40 more
   [junit4]   2> 20821 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testBasics
   [junit4]   2> 20828 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1594358524609036288,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 20828 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@5cf2ad02 commitCommandVersion:1594358524609036288
   [junit4]   2> 20859 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@459e4c95[collection1] main]
   [junit4]   2> 20860 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 20869 INFO  (searcherExecutor-85-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@459e4c95[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C10/9:delGen=1) Uninverting(_1(8.0.0):C1)))}
   [junit4]   2> 20870 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 42
   [junit4]   2> 20872 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 20874 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:human&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 20875 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Mr&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 20877 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:20&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 20882 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=name_s:Virat&wt=xml} hits=1 status=0 QTime=4
   [junit4]   2> 20883 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=multiDefault:Delhi&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 20908 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1594358524692922368,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 20908 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@5cf2ad02 commitCommandVersion:1594358524692922368
   [junit4]   2> 20918 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@35e3fa0b[collection1] main]
   [junit4]   2> 20919 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 20924 INFO  (searcherExecutor-85-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@35e3fa0b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C10/9:delGen=1) Uninverting(_2(8.0.0):C1)))}
   [junit4]   2> 20926 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 18
   [junit4]   2> 20928 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 20930 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:human&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 20931 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=cat:animal&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 20932 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Mr&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 20933 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=title:Dr&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 20934 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:20&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 20935 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=count_i:40&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 20936 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=name_s:Virat&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 20937 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=multiDefault:Delhi&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 20937 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testBasics-seed#[9269A545F6156241]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testBasics
   [junit4]   2> 20940 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testNoUniqueIdPassed-seed#[9269A545F6156241]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNoUniqueIdPassed
   [junit4]   2> 20940 INFO  (TEST-AtomicUpdateProcessorFactoryTest.testNoUniqueIdPassed-seed#[9269A545F6156241]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNoUniqueIdPassed
   [junit4]   2> 20940 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 20940 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=80198472
   [junit4]   2> 20940 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 20941 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@f23c33a: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@40d79d8d
   [junit4]   2> 20949 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 20950 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@a2108b7: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@309a8adc
   [junit4]   2> 20956 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 20956 INFO  (SUITE-AtomicUpdateProcessorFactoryTest-seed#[9269A545F6156241]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@37e4e73: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@463902aa
   [junit4]   2> 20957 INFO  (coreCloseExecutor-90-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@18e104de
   [junit4]   2> 20957 INFO  (coreCloseExecutor-90-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=417400030
   [junit4]   2> 20957 INFO  (coreCloseExecutor-90-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3ea7e34c: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@58cdf75
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.update.processor.AtomicUpdateProcessorFactoryTest_9269A545F6156241-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), title_stringNoNorms=PostingsFormat(name=Memory), title_stemmed=PostingsFormat(name=Memory), multiDefault=PostingsFormat(name=Memory), cat=PostingsFormat(name=Memory), id=FST50, text=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), title=PostingsFormat(name=Memory), title_lettertok=FST50}, docValues:{range_facet_l_dv=DocValuesFormat(name=Direct), _version_=DocValuesFormat(name=Lucene70), range_facet_i_dv=DocValuesFormat(name=Lucene70), intDvoDefault=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1346, maxMBSortInHeap=6.638173774084729, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@754ba1a7), locale=nl-BE, timezone=Africa/Nairobi
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=202978736,total=315097088
   [junit4]   2> NOTE: All tests run in this JVM: [CdcrUpdateLogTest, TestReloadAndDeleteDocs, TestSolrFieldCacheBean, LeaderElectionContextKeyTest, AtomicUpdateProcessorFactoryTest]
   [junit4] Completed [6/781 (1!)] on J2 in 1.60s, 4 tests, 1 error <<< FAILURES!

[...truncated 368 lines...]
   [junit4] Suite: org.apache.solr.core.TestJmxIntegration
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.core.TestJmxIntegration_9269A545F6156241-001/init-core-data-001
   [junit4]   2> 455180 WARN  (SUITE-TestJmxIntegration-seed#[9269A545F6156241]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=20 numCloses=20
   [junit4]   2> 455181 INFO  (SUITE-TestJmxIntegration-seed#[9269A545F6156241]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 455182 INFO  (SUITE-TestJmxIntegration-seed#[9269A545F6156241]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 455184 INFO  (SUITE-TestJmxIntegration-seed#[9269A545F6156241]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 455185 INFO  (SUITE-TestJmxIntegration-seed#[9269A545F6156241]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 455186 INFO  (SUITE-TestJmxIntegration-seed#[9269A545F6156241]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 455222 INFO  (SUITE-TestJmxIntegration-seed#[9269A545F6156241]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 455252 INFO  (SUITE-TestJmxIntegration-seed#[9269A545F6156241]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 455396 INFO  (SUITE-TestJmxIntegration-seed#[9269A545F6156241]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 455460 INFO  (SUITE-TestJmxIntegration-seed#[9269A545F6156241]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5076aee3
   [junit4]   2> 455475 INFO  (SUITE-TestJmxIntegration-seed#[9269A545F6156241]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5076aee3
   [junit4]   2> 455476 INFO  (SUITE-TestJmxIntegration-seed#[9269A545F6156241]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5076aee3
   [junit4]   2> 455478 INFO  (coreLoadExecutor-1644-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 455479 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 455519 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 455549 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 455693 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 455704 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 455704 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5076aee3
   [junit4]   2> 455704 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 455704 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.core.TestJmxIntegration_9269A545F6156241-001/init-core-data-001/]
   [junit4]   2> 455707 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=39.0537109375, floorSegmentMB=1.69140625, forceMergeDeletesPctAllowed=26.416528270888723, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 455784 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 455784 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 455793 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 455793 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 455795 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=236485851, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 455795 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@690863a2[collection1] main]
   [junit4]   2> 455796 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 455807 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 455853 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 455853 INFO  (searcherExecutor-1645-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 455853 INFO  (coreLoadExecutor-1644-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1594358980765810688
   [junit4]   2> 455854 INFO  (searcherExecutor-1645-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 455854 INFO  (searcherExecutor-1645-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 455854 INFO  (searcherExecutor-1645-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 455854 INFO  (searcherExecutor-1645-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 455855 INFO  (searcherExecutor-1645-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 455855 INFO  (searcherExecutor-1645-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 455856 INFO  (searcherExecutor-1645-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 455856 INFO  (searcherExecutor-1645-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 455860 INFO  (SUITE-TestJmxIntegration-seed#[9269A545F6156241]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 455865 INFO  (searcherExecutor-1645-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@690863a2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 455866 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[9269A545F6156241]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testJmxUpdate
   [junit4]   2> 455890 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[9269A545F6156241]) [    ] o.a.s.c.TestJmxIntegration Mbeans in server: [solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=handlerStart, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Survivor-Space,name=max, java.lang:type=Threading, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=requests, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=handlerStart, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=errors, solr:dom1=jvm,name0=os,name=systemLoadAverage, solr:dom1=core,dom2=collection1,category=QUERY,scope=/get,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=requestTimes, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=requestTimes, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/health,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=shardRequests, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.updateExecutor.running, solr:dom1=node,category=QUERY,scope=httpShardHandler,name=threadPool.httpShardExecutor.duration, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=handlerStart, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=serverErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=handlerStart, solr:dom1=jvm,name0=buffers,name1=mapped,name=TotalCapacity, solr:dom1=node,category=ADMIN,scope=/admin/health,name=errors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=totalTime, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=rollbacks, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.recoveryExecutor.submitted, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=requestTimes, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/system,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=clientErrors, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreContainerWorkExecutor.running, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=requestTimes, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=reader, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragmenter,name=gap.requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=requestTimes, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrBoundaryScanner,name=breakIterator.requests, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=totalTime, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=errors, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Survivor-Space,name=usage, solr:dom1=jvm,name0=memory,name1=pools,name2=Compressed-Class-Space,name=max, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=clientErrors, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Old-Gen,name=init, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=errors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/file,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=shardRequests, solr:dom1=core,dom2=collection1,category=TLOG,name=state, solr:dom1=core,dom2=collection1,category=CORE,scope=fs,name=spins, solr:dom1=jvm,name0=buffers,name1=direct,name=MemoryUsed, solr:dom1=jvm,name0=classes,name=unloaded, solr:dom1=jvm,name0=threads,name1=waiting,name=count, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=errors, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=threadPool.parallelCoreAdminExecutor.submitted, solr:dom1=core,dom2=collection1,category=TLOG,scope=replay,name=remaining.bytes, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=timeouts, solr:dom1=node,category=CONTAINER,scope=cores,name=loaded, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=handlerStart, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=totalTime, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=new,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=timeouts, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/config,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=handlerStart, solr:dom1=node,category=QUERY,scope=httpShardHandler,name=threadPool.httpShardExecutor.completed, solr_47374:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=clientErrors, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=serverErrors, java.lang:type=Runtime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=cumulativeDeletesById, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=cumulativeDeletesByQuery, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=timeouts, solr_49519:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=errors, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.recoveryExecutor.duration, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=timeouts, solr:dom1=core,dom2=collection1,category=CORE,scope=fs,name=totalSpace, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=handlerStart, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreContainerWorkExecutor.duration, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=clientErrors, solr:dom1=node,category=CONTAINER,scope=cores,name=unloaded, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=serverErrors, java.lang:type=MemoryPool,name=PS Eden Space, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=timeouts, solr:dom1=core,dom2=collection1,category=CORE,scope=fs,name=usableSpace, solr:dom1=jvm,name0=os,name=totalPhysicalMemorySize, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=serverErrors, solr:dom1=jvm,name0=os,name=totalSwapSpaceSize, solr:dom1=jvm,name0=memory,name1=pools,name2=Code-Cache,name=init, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=cumulativeErrors, java.lang:type=MemoryPool,name=PS Survivor Space, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=handlerStart, solr:dom1=jvm,name0=os,name=availableProcessors, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.recoveryExecutor.running, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=errors, solr:dom1=jvm,name0=memory,name1=pools,name2=Compressed-Class-Space,name=used, solr:dom1=node,category=CONTAINER,scope=fs,name=path, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=timeouts, solr:dom1=jvm,name0=threads,name=count, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=timeouts, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=caching, solr:dom1=node,category=ADMIN,scope=/admin/health,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=serverErrors, solr:dom1=jvm,name0=memory,name1=total,name=used, solr:dom1=jvm,name0=memory,name1=pools,name2=Code-Cache,name=usage, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFormatter,name=html.requests, solr:dom1=node,category=CONTAINER,scope=fs,name=spins, solr_35690:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=deletesByQuery, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=serverErrors, solr:dom1=jvm,name0=threads,name1=new,name=count, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=handlerStart, solr:dom1=core,dom2=collection1,category=TLOG,scope=replay,name=ops, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=serverErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=serverErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=timeouts, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Old-Gen,name=committed, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=totalTime, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=openedAt, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=requestTimes, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=initCount, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=errors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=handlerStart, solr:dom1=node,category=QUERY,scope=httpShardHandler,name=leasedConnections, solr:dom1=core,dom2=collection1,category=QUERY,scope=/select,name=requests, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=softAutoCommits, solr:dom1=jvm,name0=memory,name1=non-heap,name=used, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=serverErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=shardRequests, solr_38330:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=clientErrors, java.lang:type=MemoryPool,name=Code Cache, solr:dom1=core,dom2=collection1,category=CORE,name=refCount, java.nio:type=BufferPool,name=direct, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json,name=totalTime, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=errors, solr:dom1=jvm,name0=threads,name1=daemon,name=count, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=serverErrors, solr:dom1=node,category=CONTAINER,scope=fs,name=coreRoot.path, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=errors, solr:dom1=node,category=ADMIN,scope=/admin/health,name=shardRequests, solr:dom1=jvm,name0=memory,name1=non-heap,name=init, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=requestTimes, solr:dom1=core,dom2=collection1,category=CACHE,scope=searcher,name=queryResultCache, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=totalTime, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=handlerStart, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=searcherName, solr_49830:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=requests, solr_34502:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=jvm,name0=memory,name1=heap,name=max, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=shardRequests, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreLoadExecutor.duration, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=shardRequests, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=serverErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=timeouts, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=errors, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=requestTimes, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/file,name=timeouts, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=indexSize, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=errors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=shardRequests, solr:dom1=jvm,name0=buffers,na
   [junit4]   2> me1=mapped,name=Count, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=errors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/file,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=serverErrors, java.lang:type=Compilation, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.updateExecutor.submitted, solr:dom1=jvm,name0=memory,name1=pools,name2=Compressed-Class-Space,name=committed, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=requests, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=serverErrors, solr:dom1=core,dom2=collection1,category=TLOG,scope=buffered,name=ops, solr_40772:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=adds, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=errors, solr:dom1=jvm,name0=os,name=version, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=serverErrors, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=pendingConnections, solr:dom1=jvm,name0=memory,name1=pools,name2=Metaspace,name=usage, solr:dom1=core,dom2=collection1,category=CACHE,scope=searcher,name=documentCache, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=shardRequests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/plugins,name=totalTime, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=handlerStart, java.lang:type=MemoryPool,name=Compressed Class Space, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=requestTimes, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=requests, solr:dom1=jvm,name0=os,name=committedVirtualMemorySize, solr:dom1=core,dom2=collection1,category=QUERY,scope=/tvrh,name=errors, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Old-Gen,name=usage, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=requests, solr:dom1=jvm,name0=memory,name1=non-heap,name=max, solr:dom1=jvm,name0=os,name=maxFileDescriptorCount, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragmentsBuilder,name=simple.requests, solr:dom1=core,dom2=collection1,category=CORE,name=coreName, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreLoadExecutor.running, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=handlerStart, solr:dom1=jvm,name0=memory,name1=heap,name=used, solr:dom1=core,dom2=collection1,category=CACHE,scope=searcher,name=perSegFilter, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/health,name=totalTime, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=errors, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=confFilesToReplicate, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=totalTime, solr:dom1=node,category=CONTAINER,scope=fs,name=coreRoot.usableSpace, solr:dom1=jvm,name0=os,name=systemCpuLoad, solr_35405:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragListBuilder,name=default.requests, solr:dom1=node,category=ADMIN,scope=/admin/info,name=shardRequests, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=splits, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=handlerStart, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=fetcher, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=requestTimes, solr:dom1=jvm,name0=threads,name1=blocked,name=count, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=requestTimes, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=shardRequests, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=totalTime, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrBoundaryScanner,name=default.requests, solr:dom1=jvm,name0=gc,name1=PS-MarkSweep,name=time, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=totalTime, solr:dom1=core,dom2=collection1,category=TLOG,scope=applyingBuffered,name=ops, solr:dom1=node,category=ADMIN,scope=/admin/info,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=totalTime, solr:dom1=jvm,name0=buffers,name1=direct,name=TotalCapacity, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=serverErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=handlerStart, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=generation, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=errors, java.lang:type=OperatingSystem, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/tvrh,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=clientErrors, solr:dom1=jvm,name0=threads,name1=terminated,name=count, java.lang:type=MemoryManager,name=Metaspace Manager, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=handlerStart, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=shardRequests, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=totalTime, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=errors, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=handlerStart, solr:dom1=node,category=ADMIN,scope=/admin/health,name=requests, solr:dom1=core,dom2=collection1,category=TLOG,scope=replay,name=remaining.logs, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/system,name=requestTimes, solr:dom1=core,dom2=collection1,category=CORE,scope=fs,name=path, solr:dom1=node,category=CONTAIN

[...truncated too long message...]


   [junit4]   2> 1399614 INFO  (coreCloseExecutor-4369-thread-1) [n:127.0.0.1:51053_solr c:outOfSyncReplicasCannotBecomeLeader-false s:shard1 r:core_node64 x:outOfSyncReplicasCannotBecomeLeader-false_shard1_replica_n63] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.outOfSyncReplicasCannotBecomeLeader-false.shard1.leader, tag=977288414
   [junit4]   2> 1399617 INFO  (coreCloseExecutor-4366-thread-1) [n:127.0.0.1:36383_solr c:outOfSyncReplicasCannotBecomeLeader-false s:shard1 r:core_node66 x:outOfSyncReplicasCannotBecomeLeader-false_shard1_replica_n65] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.outOfSyncReplicasCannotBecomeLeader-false.shard1.leader, tag=1683673221
   [junit4]   2> 1399618 INFO  (zkCallback-2448-thread-1-processing-n:127.0.0.1:36383_solr) [n:127.0.0.1:36383_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36383_solr
   [junit4]   2> 1399619 INFO  (jetty-closer-2325-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5596de9e{/solr,null,UNAVAILABLE}
   [junit4]   2> 1399620 INFO  (jetty-closer-2325-thread-2) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1399624 INFO  (jetty-closer-2325-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@474f539c{/solr,null,UNAVAILABLE}
   [junit4]   2> 1399624 INFO  (jetty-closer-2325-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@41e39222{/solr,null,UNAVAILABLE}
   [junit4]   2> 1399626 INFO  (jetty-closer-2325-thread-3) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1399626 INFO  (jetty-closer-2325-thread-1) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1399627 ERROR (SUITE-TestCloudConsistency-seed#[9269A545F6156241]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1399627 INFO  (SUITE-TestCloudConsistency-seed#[9269A545F6156241]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58298 58298
   [junit4]   2> 1399780 INFO  (Thread-2239) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58298 58298
   [junit4]   2> 1399781 WARN  (Thread-2239) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	60	/solr/collections/outOfSyncReplicasCannotBecomeLeader-false/terms/shard1
   [junit4]   2> 	51	/solr/collections/outOfSyncReplicasCannotBecomeLeader-true/terms/shard1
   [junit4]   2> 	13	/solr/aliases.json
   [junit4]   2> 	12	/solr/security.json
   [junit4]   2> 	7	/solr/configs/outOfSyncReplicasCannotBecomeLeader-false.AUTOCREATED/managed-schema
   [junit4]   2> 	7	/solr/collections/outOfSyncReplicasCannotBecomeLeader-false/collectionprops.json
   [junit4]   2> 	7	/solr/configs/outOfSyncReplicasCannotBecomeLeader-false.AUTOCREATED
   [junit4]   2> 	6	/solr/collections/outOfSyncReplicasCannotBecomeLeader-true/collectionprops.json
   [junit4]   2> 	6	/solr/configs/outOfSyncReplicasCannotBecomeLeader-true.AUTOCREATED
   [junit4]   2> 	6	/solr/configs/outOfSyncReplicasCannotBecomeLeader-true.AUTOCREATED/managed-schema
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	97	/solr/collections/outOfSyncReplicasCannotBecomeLeader-false/state.json
   [junit4]   2> 	78	/solr/collections/outOfSyncReplicasCannotBecomeLeader-true/state.json
   [junit4]   2> 	13	/solr/clusterprops.json
   [junit4]   2> 	13	/solr/clusterstate.json
   [junit4]   2> 	6	/solr/autoscaling.json
   [junit4]   2> 	2	/solr/collections/outOfSyncReplicasCannotBecomeLeader-true/leader_elect/shard1/election/73417596525871133-core_node62-n_0000000000
   [junit4]   2> 	2	/solr/overseer_elect/election/73417596525871112-127.0.0.1:33467_solr-n_0000000000
   [junit4]   2> 	2	/solr/overseer_elect/election/73417596525871113-127.0.0.1:44364_solr-n_0000000002
   [junit4]   2> 	2	/solr/collections/outOfSyncReplicasCannotBecomeLeader-false/leader_elect/shard1/election/73417596525871133-core_node62-n_0000000009
   [junit4]   2> 	2	/solr/overseer_elect/election/73417596525871130-127.0.0.1:53326_solr-n_0000000007
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	13	/solr/live_nodes
   [junit4]   2> 	13	/solr/collections
   [junit4]   2> 	6	/solr/overseer/queue
   [junit4]   2> 	6	/solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 	6	/solr/overseer/queue-work
   [junit4]   2> 	6	/solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudConsistency_9269A545F6156241-001
   [junit4]   2> Mar 08, 2018 9:09:14 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1687, maxMBSortInHeap=5.511789556878046, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@1357099c), locale=it-CH, timezone=Etc/GMT+3
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=181027600,total=524288000
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsLockFactoryTest, TestConfigSetImmutable, TestInitQParser, DistributedQueueTest, SolrIndexConfigTest, TestSolrXml, TestUseDocValuesAsStored2, TestQuerySenderNoQuery, OverseerRolesTest, CursorPagingTest, TestAddFieldRealTimeGet, ExternalFileFieldSortTest, ActionThrottleTest, MetricsConfigTest, DistributedExpandComponentTest, EchoParamsTest, DefaultValueUpdateProcessorTest, TestGeoJSONResponseWriter, TestTestInjection, FieldAnalysisRequestHandlerTest, TestCSVLoader, SuggesterWFSTTest, ResponseHeaderTest, HttpPartitionTest, TestJsonFacetRefinement, HdfsBasicDistributedZkTest, ForceLeaderTest, TestUnifiedSolrHighlighter, TestSha256AuthenticationProvider, TestWordDelimiterFilterFactory, BaseCdcrDistributedZkTest, TestOrdValues, HdfsTlogReplayBufferedWhileIndexingTest, TestNumericTerms32, TestPHPSerializedResponseWriter, TestInitParams, ChaosMonkeySafeLeaderWithPullReplicasTest, ReplicaListTransformerTest, TestOverriddenPrefixQueryForCustomFieldType, HdfsAutoAddReplicasIntegrationTest, TestGroupingSearch, TestFreeTextSuggestions, TestSchemaResource, TestConfigSetsAPI, TestApiFramework, TestSolrCloudWithHadoopAuthPlugin, RequestHandlersTest, CollectionsAPIAsyncDistributedZkTest, TestJoin, PluginInfoTest, SolrMetricReporterTest, TestTlogReplica, CdcrVersionReplicationTest, BlockJoinFacetDistribTest, TestSolrDeletionPolicy1, TestDocSet, OverriddenZkACLAndCredentialsProvidersTest, SolrGraphiteReporterTest, TestUtilizeNode, CachingDirectoryFactoryTest, MergeStrategyTest, ClassificationUpdateProcessorTest, TestLuceneMatchVersion, PeerSyncReplicationTest, InfoHandlerTest, V2ApiIntegrationTest, ReplaceNodeNoTargetTest, TestWriterPerf, ShardRoutingTest, RulesTest, TestCryptoKeys, TestShortCircuitedRequests, ZkNodePropsTest, TestComplexPhraseQParserPlugin, BlockDirectoryTest, TestTrackingShardHandlerFactory, TestLuceneIndexBackCompat, TestLegacyFieldReuse, BasicAuthIntegrationTest, TestBinaryResponseWriter, BadIndexSchemaTest, VMParamsZkACLAndCredentialsProvidersTest, TestRemoteStreaming, QueryResultKeyTest, SolrPluginUtilsTest, CdcrReplicationDistributedZkTest, TestInPlaceUpdatesDistrib, TestStressCloudBlindAtomicUpdates, OpenCloseCoreStressTest, DistributedQueryElevationComponentTest, TestCloudRecovery, TestRestoreCore, AutoScalingHandlerTest, UpdateParamsTest, TestMergePolicyConfig, DateRangeFieldTest, TestSubQueryTransformer, TimeRoutedAliasUpdateProcessorTest, DebugComponentTest, TestMultiValuedNumericRangeQuery, OutputWriterTest, TestPushWriter, TestLMDirichletSimilarityFactory, TestCoreAdminApis, LegacyCloudClusterPropTest, TestCloudConsistency]
   [junit4] Completed [304/781 (4!)] on J0 in 67.17s, 2 tests, 1 failure <<< FAILURES!

[...truncated 9094 lines...]
   [junit4] Suite: org.apache.solr.ltr.TestLTRReRankingPipeline
   [junit4]   2> 20455 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[369054BF4F2D0802]) [    ] o.a.s.l.TestLTRReRankingPipeline rerank 1 documents 
   [junit4]   2> 20462 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[369054BF4F2D0802]) [    ] o.a.s.l.TestLTRReRankingPipeline doc 0 in pos 0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTRReRankingPipeline -Dtests.method=testDifferentTopN -Dtests.seed=369054BF4F2D0802 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=pt-PT -Dtests.timezone=Antarctica/Mawson -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.20s J0 | TestLTRReRankingPipeline.testDifferentTopN <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1.0> but was:<0.0>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([369054BF4F2D0802:C73126EF7A96C290]:0)
   [junit4]    > 	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@5f6cf3b5), locale=pt-PT, timezone=Antarctica/Mawson
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=249376480,total=337117184
   [junit4]   2> NOTE: All tests run in this JVM: [TestModelManagerPersistence, TestLTRWithSort, TestRankingFeature, TestLTRReRankingPipeline]
   [junit4] Completed [15/37 (1!)] on J0 in 0.27s, 3 tests, 1 failure <<< FAILURES!

[...truncated 41996 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 7 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/7/

7 tests failed.
FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.testNormalMove

Error Message:
Error from server at http://127.0.0.1:33984/solr: delete the collection time out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33984/solr: delete the collection time out:180s
	at __randomizedtesting.SeedInfo.seed([EE0FBB5E5335ED21:48D70CF9D6D42F3B]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.MiniSolrCloudCluster.deleteAllCollections(MiniSolrCloudCluster.java:451)
	at org.apache.solr.cloud.MoveReplicaTest.beforeTest(MoveReplicaTest.java:97)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:968)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.testNormalFailedMove

Error Message:
Error from server at http://127.0.0.1:33984/solr: delete the collection time out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33984/solr: delete the collection time out:180s
	at __randomizedtesting.SeedInfo.seed([EE0FBB5E5335ED21:C904A4167F777338]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.MiniSolrCloudCluster.deleteAllCollections(MiniSolrCloudCluster.java:451)
	at org.apache.solr.cloud.MoveReplicaTest.beforeTest(MoveReplicaTest.java:97)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:968)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([EE0FBB5E5335ED21:44C268ACE4E638F1]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertFalse(Assert.java:68)
	at org.junit.Assert.assertFalse(Assert.java:79)
	at org.apache.solr.cloud.MoveReplicaTest.testFailedMove(MoveReplicaTest.java:303)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN

Error Message:
expected:<1.0> but was:<0.0>

Stack Trace:
java.lang.AssertionError: expected:<1.0> but was:<0.0>
	at __randomizedtesting.SeedInfo.seed([251D9C63125482E8:D4BCEE3327EF487A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:443)
	at org.junit.Assert.assertEquals(Assert.java:512)
	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ZkControllerTest.testPublishAndWaitForDownStates

Error Message:
The ZkController.publishAndWaitForDownStates should have timed out but it didn't

Stack Trace:
java.lang.AssertionError: The ZkController.publishAndWaitForDownStates should have timed out but it didn't
	at __randomizedtesting.SeedInfo.seed([EE0FBB5E5335ED21:C900D7B2F2A2A21E]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.ZkControllerTest.testPublishAndWaitForDownStates(ZkControllerTest.java:306)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload

Error Message:
Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>

Stack Trace:
java.lang.AssertionError: Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>
	at __randomizedtesting.SeedInfo.seed([EE0FBB5E5335ED21:ECB66032207631EC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload(TestJmxIntegration.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart

Error Message:
IOException occured when talking to server at: http://127.0.0.1:46032/solr/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:46032/solr/collection1
	at __randomizedtesting.SeedInfo.seed([EE0FBB5E5335ED21:36F87FBAF8EE2F7D]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
	at org.apache.solr.handler.TestReplicationHandler.index(TestReplicationHandler.java:180)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart(TestReplicationHandler.java:643)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:46032 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 47 more




Build Log:
[...truncated 14 lines...]
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://git.apache.org/lucene-solr.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:862)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
	at hudson.scm.SCM.checkout(SCM.java:495)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1724)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress git://git.apache.org/lucene-solr.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: fatal: unable to connect to git.apache.org:
git.apache.org[0: 54.84.58.65]: errno=Connection refused


	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1996)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1715)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:405)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
	at hudson.remoting.UserRequest.perform(UserRequest.java:207)
	at hudson.remoting.UserRequest.perform(UserRequest.java:53)
	at hudson.remoting.Request$2.run(Request.java:358)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
		at hudson.remoting.UserResponse.retrieve(UserRequest.java:310)
		at hudson.remoting.Channel.call(Channel.java:908)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
		at sun.reflect.GeneratedMethodAccessor1042.invoke(Unknown Source)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
		at com.sun.proxy.$Proxy111.execute(Unknown Source)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:860)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
		at hudson.scm.SCM.checkout(SCM.java:495)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1724)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Fetching upstream changes from git://git.apache.org/lucene-solr.git
 > git --version # timeout=10
 > git fetch --tags --progress git://git.apache.org/lucene-solr.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 1b8651db4f2656fdaf59dd0f44550766150ed829 (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1b8651db4f2656fdaf59dd0f44550766150ed829
Commit message: "SOLR-12011: Remove unused import"
 > git rev-list --no-walk 4bdc99a465f830c5fe71abc2833327b9c27b221c # timeout=10
No emails were triggered.
[Lucene-Solr-BadApples-Tests-master] $ /home/jenkins/tools/ant/apache-ant-1.8.4/bin/ant jenkins-badapples
Buildfile: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/build.xml

jenkins-badapples:

-print-java-info:
[java-info] java version "1.8.0_144"
[java-info] Java(TM) SE Runtime Environment (1.8.0_144-b01, Oracle Corporation)
[java-info] Java HotSpot(TM) 64-Bit Server VM (25.144-b01, Oracle Corporation)
[java-info] Test args: []

clean:

clean:

clean:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: Apache Ivy 2.4.0 - 20141213170938 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve-groovy:
[ivy:cachepath] :: resolving dependencies :: org.codehaus.groovy#groovy-all-caller;working
[ivy:cachepath] 	confs: [default]
[ivy:cachepath] 	found org.codehaus.groovy#groovy-all;2.4.13 in public
[ivy:cachepath] :: resolution report :: resolve 328ms :: artifacts dl 3ms
	---------------------------------------------------------------------
	|                  |            modules            ||   artifacts   |
	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
	---------------------------------------------------------------------
	|      default     |   1   |   0   |   0   |   0   ||   1   |   0   |
	---------------------------------------------------------------------

-test-with-heapdumps-enabled:
     [echo] Java HotSpot(TM) 64-Bit Server VM: Enabling heap dumps on OutOfMemoryError to dir '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/heapdumps'.
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/heapdumps

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve-groovy:

test:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

-clover.load:

resolve-groovy:

-init-totals:

test-core:

-clover.disable:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

-clover.load:

-clover.classpath:

-clover.setup:

clover:

-check-git-state:

-git-cleanroot:

-copy-git-state:
     [copy] Copying 1 file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build

git-autoclean:

resolve:

init:

compile-core:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java
    [javac] Compiling 828 source files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/util/graph/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/document/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/util/bkd/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/compressing/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/lucene60/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/util/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/index/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/util/fst/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/analysis/tokenattributes/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/search/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/util/packed/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/util/mutable/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/lucene70/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/perfield/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/analysis/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/analysis/standard/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/store/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/blocktree/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/search/spans/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/search/similarities/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/util/automaton/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/lucene50/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java/org/apache/lucene/geo/package-info.class
     [copy] Copying 3 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java

compile-test-framework:

-check-git-state:

-git-cleanroot:

-copy-git-state:

git-autoclean:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

init:

compile-lucene-core:

-check-git-state:

-git-cleanroot:

-copy-git-state:

git-autoclean:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

init:

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

compile-core:

compile-codecs:

-check-git-state:

-git-cleanroot:

-copy-git-state:

git-autoclean:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

common.init:

compile-lucene-core:

init:

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

compile-core:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/codecs/classes/java
    [javac] Compiling 67 source files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/codecs/classes/java
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/bloom/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/memory/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/simpletext/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/blockterms/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/blocktreeords/package-info.class
     [copy] Copying 3 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/codecs/classes/java

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

common.compile-core:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/test-framework/classes/java
    [javac] Compiling 191 source files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/test-framework/classes/java
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/asserting/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/ramonly/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/test-framework/classes/java/org/apache/lucene/mockfile/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/cheapbastard/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/mockrandom/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/cranky/package-info.class
    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/compressing/dummy/package-info.class
     [copy] Copying 4 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/test-framework/classes/java

compile-core:

compile-test:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/test
    [javac] Compiling 496 source files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/test
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
     [copy] Copying 3 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/test

-mrjar-classes-uptodate:

resolve-groovy:

patch-mrjar-classes:
[patch-cls] Remapped: org/apache/lucene/analysis/tokenattributes/CharTermAttributeImpl
[patch-cls] Remapped: org/apache/lucene/codecs/compressing/LZ4
[patch-cls] Remapped: org/apache/lucene/document/BinaryPoint$2
[patch-cls] Remapped: org/apache/lucene/document/DoubleRange
[patch-cls] Remapped: org/apache/lucene/document/FloatRange
[patch-cls] Remapped: org/apache/lucene/document/IntRange
[patch-cls] Remapped: org/apache/lucene/document/LongRange
[patch-cls] Remapped: org/apache/lucene/index/BitsSlice
[patch-cls] Remapped: org/apache/lucene/index/CodecReader
[patch-cls] Remapped: org/apache/lucene/index/MergeReaderWrapper
[patch-cls] Remapped: org/apache/lucene/search/BooleanScorer$TailPriorityQueue
[patch-cls] Remapped: org/apache/lucene/util/BytesRef
[patch-cls] Remapped: org/apache/lucene/util/BytesRefArray
[patch-cls] Remapped: org/apache/lucene/util/CharsRef$UTF16SortedAsUTF8Comparator
[patch-cls] Remapped: org/apache/lucene/util/CharsRef
[patch-cls] Remapped: org/apache/lucene/util/IntsRef
[patch-cls] Remapped: org/apache/lucene/util/LongsRef
[patch-cls] Remapped: org/apache/lucene/util/StringHelper
[patch-cls] Remapped: org/apache/lucene/util/automaton/Automaton$Builder
[patch-cls] Remapped: org/apache/lucene/util/automaton/Automaton
[patch-cls] Remapped 20 class files for Java 9 to: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/classes/java9
    [touch] Creating /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/patch-mrjar.stamp

install-junit4-taskdef:

validate:

-init-totals:

-test:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/test
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/build/core/test/temp
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/.caches/test-stats/core
   [junit4] <JUnit4> says kaixo! Master seed: 135AC8A5366D34D8
   [junit4] Executing 486 suites with 3 JVMs.
   [junit4] 
   [junit4] Started J1 PID(2376@localhost).
   [junit4] Started J0 PID(2379@localhost).
   [junit4] Started J2 PID(2383@localhost).
   [junit4] Suite: org.apache.lucene.index.TestSegmentTermEnum
   [junit4] Completed [1/486] on J2 in 1.51s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterOnJRECrash
   [junit4] IGNOR/A 0.01s J2 | TestIndexWriterOnJRECrash.testNRTThreads
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed [2/486] on J2 in 0.04s, 1 test, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestOmitPositions
   [junit4] Completed [3/486] on J1 in 2.30s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDocCount
   [junit4] Completed [4/486] on J1 in 0.78s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDocValues
   [junit4] Completed [5/486] on J1 in 0.49s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPrefixCodedTerms
   [junit4] Completed [6/486] on J1 in 0.46s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.geo.TestGeoEncodingUtils
   [junit4] Completed [7/486] on J1 in 1.41s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNRTReaderCleanup
   [junit4] Completed [8/486] on J1 in 0.08s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestDeterminism
   [junit4] Completed [9/486] on J1 in 1.82s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.TestSearch
   [junit4] Completed [10/486] on J1 in 0.29s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestRecyclingByteBlockAllocator
   [junit4] Completed [11/486] on J1 in 0.08s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestParallelReaderEmptyIndex
   [junit4] Completed [12/486] on J1 in 0.16s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterOutOfFileDescriptors
   [junit4] Completed [13/486] on J2 in 9.91s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestAtomicUpdate
   [junit4] Completed [14/486] on J1 in 5.04s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestLockFactory
   [junit4] Completed [15/486] on J1 in 0.05s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTerms
   [junit4] Completed [16/486] on J2 in 1.72s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTermVectors
   [junit4] Completed [17/486] on J1 in 1.83s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestBinaryDocValuesUpdates
   [junit4] IGNOR/A 0.00s J2 | TestBinaryDocValuesUpdates.testTonsOfUpdates
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed [18/486] on J2 in 10.96s, 29 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestSameTokenSamePosition
   [junit4] Completed [19/486] on J2 in 0.09s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.FiniteStringsIteratorTest
   [junit4] Completed [20/486] on J2 in 1.39s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestStopFilter
   [junit4] Completed [21/486] on J2 in 0.09s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestRoaringDocIdSet
   [junit4] Completed [22/486] on J2 in 1.78s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTermdocPerf
   [junit4] Completed [23/486] on J2 in 0.06s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexFileDeleter
   [junit4] Completed [24/486] on J2 in 1.76s, 10 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestCharsRef
   [junit4] Completed [25/486] on J2 in 0.30s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestComplexExplanationsOfNonMatches
   [junit4] Completed [26/486] on J2 in 0.57s, 21 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestFilterDirectory
   [junit4] IGNOR/A 0.00s J2 | TestFilterDirectory.testPendingDeletions
   [junit4]    > Assumption #1: we can only install VirusCheckingFS on an FSDirectory
   [junit4] IGNOR/A 0.00s J2 | TestFilterDirectory.testFsyncDoesntCreateNewFiles
   [junit4]    > Assumption #1: test only works for FSDirectory subclasses
   [junit4] Completed [27/486] on J2 in 1.33s, 44 tests, 2 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.standard.TestStandardAnalyzer
   [junit4] Completed [28/486] on J0 in 32.38s, 36 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestDisjunctionMaxQuery
   [junit4] Completed [29/486] on J2 in 1.68s, 12 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexReaderClose
   [junit4] Completed [30/486] on J0 in 1.91s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPerSegmentDeletes
   [junit4] Completed [31/486] on J0 in 0.78s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.Test2BSortedDocValuesFixedSorted
   [junit4] IGNOR/A 0.01s J0 | Test2BSortedDocValuesFixedSorted.testFixedSorted
   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=Takes ~ 6 hours if the heap is 5gb))
   [junit4] Completed [32/486] on J0 in 0.03s, 1 test, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestFixedBitSet
   [junit4] Completed [33/486] on J2 in 2.73s, 19 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSloppyPhraseQuery2
   [junit4] Completed [34/486] on J2 in 4.38s, 9 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestNeedsScores
   [junit4] Completed [35/486] on J2 in 0.06s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestScorerPerf
   [junit4] Completed [36/486] on J2 in 7.28s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestMixedCodecs
   [junit4] Completed [37/486] on J2 in 2.51s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestBoolean2ScorerSupplier
   [junit4] Completed [38/486] on J2 in 0.44s, 10 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestBytesRefHash
   [junit4] Completed [39/486] on J2 in 2.41s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestCloseableThreadLocal
   [junit4] Completed [40/486] on J2 in 0.06s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpanBoostQuery
   [junit4] Completed [41/486] on J2 in 0.03s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestCompiledAutomaton
   [junit4] Completed [42/486] on J2 in 0.94s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestSetOnce
   [junit4] Completed [43/486] on J2 in 0.20s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestNamedSPILoader
   [junit4] Completed [44/486] on J2 in 0.06s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestFieldsReader
   [junit4] Completed [45/486] on J2 in 0.74s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestInfoStream
   [junit4] Completed [46/486] on J2 in 0.07s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestSwappedIndexFiles
   [junit4] Completed [47/486] on J2 in 0.25s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestFilterCodecReader
   [junit4] Completed [48/486] on J2 in 0.22s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestCharacterUtils
   [junit4] Completed [49/486] on J2 in 0.04s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestParallelCompositeReader
   [junit4] Completed [50/486] on J2 in 0.66s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.Test2BPoints
   [junit4] IGNOR/A 0.00s J2 | Test2BPoints.test2D
   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=takes at least 4 hours and consumes many GB of temp disk space))
   [junit4] IGNOR/A 0.00s J2 | Test2BPoints.test1D
   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=takes at least 4 hours and consumes many GB of temp disk space))
   [junit4] Completed [51/486] on J2 in 0.01s, 2 tests, 2 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.similarities.TestDistributionSPL
   [junit4] Completed [52/486] on J1 in 44.42s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestArrayUtil
   [junit4] Completed [53/486] on J2 in 5.96s, 13 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestOperations
   [junit4] Completed [54/486] on J2 in 2.38s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTieredMergePolicy
   [junit4] Completed [55/486] on J1 in 8.70s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestSingleInstanceLockFactory
   [junit4] Completed [56/486] on J1 in 2.53s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestMultiTermConstantScore
   [junit4] Completed [57/486] on J1 in 2.19s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNRTThreads
   [junit4] Completed [58/486] on J2 in 8.60s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestSmallFloat
   [junit4] Completed [59/486] on J2 in 0.40s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.tokenattributes.TestPackedTokenAttributeImpl
   [junit4] Completed [60/486] on J2 in 0.05s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestNIOFSDirectory
   [junit4] Completed [61/486] on J2 in 2.27s, 42 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestUsageTrackingFilterCachingPolicy
   [junit4] Completed [62/486] on J2 in 0.06s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestComplexExplanations
   [junit4] Completed [63/486] on J1 in 4.48s, 21 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestFilterWeight
   [junit4] Completed [64/486] on J1 in 0.04s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestForTooMuchCloning
   [junit4] Completed [65/486] on J1 in 0.49s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestSumDocFreq
   [junit4] Completed [66/486] on J1 in 0.50s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestFileSwitchDirectory
   [junit4] IGNOR/A 0.02s J1 | TestFileSwitchDirectory.testPendingDeletions
   [junit4]    > Assumption #1: we can only install VirusCheckingFS on an FSDirectory
   [junit4] IGNOR/A 0.01s J1 | TestFileSwitchDirectory.testFsyncDoesntCreateNewFiles
   [junit4]    > Assumption #1: test only works for FSDirectory subclasses
   [junit4] Completed [67/486] on J1 in 2.06s, 43 tests, 2 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestRAMDirectory
   [junit4] IGNOR/A 0.01s J1 | TestRAMDirectory.testFsyncDoesntCreateNewFiles
   [junit4]    > Assumption #1: test only works for FSDirectory subclasses
   [junit4] IGNOR/A 0.00s J1 | TestRAMDirectory.testPendingDeletions
   [junit4]    > Assumption #1: we can only install VirusCheckingFS on an FSDirectory
   [junit4] Completed [68/486] on J1 in 1.49s, 46 tests, 2 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestStringMSBRadixSorter
   [junit4] Completed [69/486] on J2 in 6.35s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.similarities.TestSimilarityBase
   [junit4] Completed [70/486] on J2 in 0.83s, 27 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSimpleExplanationsWithFillerDocs
   [junit4] Completed [71/486] on J0 in 48.38s, 70 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.TestDemo
   [junit4] Completed [72/486] on J0 in 0.24s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestCrashCausesCorruptIndex
   [junit4] Completed [73/486] on J0 in 0.71s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestMultiTermsEnum
   [junit4] Completed [74/486] on J0 in 0.05s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpanNotQuery
   [junit4] Completed [75/486] on J0 in 0.15s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTimeLimitingCollector
   [junit4] Completed [76/486] on J1 in 5.31s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestMultiMMap
   [junit4] Completed [77/486] on J2 in 4.10s, 54 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestApproximationSearchEquivalence
   [junit4] Completed [78/486] on J2 in 2.00s, 10 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.TestAssertions
   [junit4] Completed [79/486] on J2 in 0.04s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestMultiCollector
   [junit4] Completed [80/486] on J2 in 0.50s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterOnDiskFull
   [junit4] Completed [81/486] on J2 in 1.21s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestLevenshteinAutomata
   [junit4] Completed [82/486] on J1 in 4.36s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestStringHelper
   [junit4] Completed [83/486] on J1 in 0.06s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestLSBRadixSorter
   [junit4] Completed [84/486] on J1 in 0.22s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestAddIndexes
   [junit4] Completed [85/486] on J1 in 5.04s, 25 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestShardSearching
   [junit4] Completed [86/486] on J2 in 10.58s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDocumentsWriterStallControl
   [junit4] Completed [87/486] on J1 in 10.06s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestParallelLeafReader
   [junit4] Completed [88/486] on J1 in 0.63s, 10 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestAllFilesHaveChecksumFooter
   [junit4] Completed [89/486] on J1 in 1.78s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriter
   [junit4] Completed [90/486] on J0 in 25.50s, 75 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.TestMergeSchedulerExternal
   [junit4] Completed [91/486] on J1 in 0.84s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterOnVMError
   [junit4] IGNOR/A 0.00s J1 | TestIndexWriterOnVMError.testCheckpoint
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed [92/486] on J1 in 1.08s, 3 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene70.TestLucene70SegmentInfoFormat
   [junit4] Completed [93/486] on J1 in 1.14s, 16 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestBufferedChecksum
   [junit4] Completed [94/486] on J1 in 0.11s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestDeterminizeLexicon
   [junit4] Completed [95/486] on J0 in 5.19s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestBlendedTermQuery
   [junit4] Completed [96/486] on J0 in 0.05s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestMinimize
   [junit4] Completed [97/486] on J1 in 3.52s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNumericDocValuesUpdates
   [junit4] Completed [98/486] on J2 in 23.03s, 32 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.document.TestIntRange
   [junit4] Completed [99/486] on J2 in 0.03s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.packed.TestDirectPacked
   [junit4] Completed [100/486] on J2 in 1.54s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestExitableDirectoryReader
   [junit4] Completed [101/486] on J2 in 0.01s, 1 test, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestRamUsageEstimator
   [junit4] IGNOR/A 0.01s J2 | TestRamUsageEstimator.testPrintValues
   [junit4]    > Assumption #1: Specify -Dtests.verbose=true to print constants of RamUsageEstimator.
   [junit4] Completed [102/486] on J2 in 0.04s, 5 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.similarities.TestSimilarity2
   [junit4] Completed [103/486] on J2 in 0.36s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterExceptions2
   [junit4] Completed [104/486] on J1 in 12.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTopDocsMerge
   [junit4] Completed [105/486] on J2 in 1.79s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.graph.TestGraphTokenStreamFiniteStrings
   [junit4] Completed [106/486] on J2 in 0.17s, 12 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpanExplanationsOfNonMatches
   [junit4] Completed [107/486] on J2 in 0.13s, 30 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestFixedLengthBytesRefArray
   [junit4] Completed [108/486] on J2 in 0.11s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpanNearQuery
   [junit4] Completed [109/486] on J2 in 0.08s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPayloadsOnVectors
   [junit4] Completed [110/486] on J2 in 0.10s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexOptions
   [junit4] Completed [111/486] on J2 in 1.90s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.geo.TestGeoUtils
   [junit4] Completed [112/486] on J2 in 7.67s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestAllFilesHaveCodecHeader
   [junit4] Completed [113/486] on J2 in 0.39s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestDocValuesRewriteMethod
   [junit4] Completed [114/486] on J2 in 4.17s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.similarities.TestIndependenceSaturated
   [junit4] Completed [115/486] on J0 in 28.06s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestMultiPhraseEnum
   [junit4] Completed [116/486] on J0 in 0.09s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.document.TestLongRange
   [junit4] Completed [117/486] on J0 in 0.06s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.geo.TestPolygon
   [junit4] Completed [118/486] on J0 in 0.20s, 15 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpanCollection
   [junit4] Completed [119/486] on J0 in 0.12s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestFilterLeafReader
   [junit4] Completed [120/486] on J0 in 0.10s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestOmitNorms
   [junit4] Completed [121/486] on J0 in 2.11s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestTimSorterWorstCase
   [junit4] Completed [122/486] on J1 in 19.66s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestCachingTokenFilter
   [junit4] Completed [123/486] on J1 in 0.19s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpanContainQuery
   [junit4] Completed [124/486] on J1 in 0.13s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.perfield.TestPerFieldPostingsFormat2
   [junit4] Completed [125/486] on J2 in 7.81s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.compressing.TestHighCompressionMode
   [junit4] Completed [126/486] on J2 in 2.11s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTermRangeQuery
   [junit4] Completed [127/486] on J2 in 0.25s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.compressing.TestFastCompressionMode
   [junit4] Completed [128/486] on J2 in 1.57s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestPrefixQuery
   [junit4] Completed [129/486] on J1 in 9.47s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTermScorer
   [junit4] Completed [130/486] on J2 in 3.40s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.document.TestDoubleRange
   [junit4] Completed [131/486] on J2 in 0.03s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestRollingUpdates
   [junit4] Completed [132/486] on J2 in 2.01s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.TestAutomaton
   [junit4] Completed [133/486] on J1 in 5.56s, 56 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSortedSetSortField
   [junit4] Completed [134/486] on J1 in 0.32s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestFilterSpans
   [junit4] Completed [135/486] on J1 in 0.05s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestStressDeletes
   [junit4] Completed [136/486] on J1 in 2.30s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestBooleanScorer
   [junit4] Completed [137/486] on J2 in 5.04s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestSpanTermQuery
   [junit4] Completed [138/486] on J2 in 0.04s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.TestExternalCodecs
   [junit4] Completed [139/486] on J1 in 1.09s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestBooleanQueryVisitSubscorers
   [junit4] Completed [140/486] on J1 in 0.20s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDoc
   [junit4] Completed [141/486] on J1 in 0.44s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestMatchAllDocsQuery
   [junit4] Completed [142/486] on J1 in 0.07s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterLockRelease
   [junit4] Completed [143/486] on J1 in 0.05s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene50.TestBlockPostingsFormat3
   [junit4] Completed [144/486] on J2 in 3.13s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestStressAdvance
   [junit4] Completed [145/486] on J2 in 5.27s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestMergedIterator
   [junit4] Completed [146/486] on J2 in 0.86s, 25 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.perfield.TestPerFieldPostingsFormat
   [junit4] IGNOR/A 0.00s J0 | TestPerFieldPostingsFormat.testPostingsEnumReuse
   [junit4]    > Assumption #1: The MockRandom PF randomizes content on the fly, so we can't check it
   [junit4] IGNOR/A 0.16s J0 | TestPerFieldPostingsFormat.testMergeStability
   [junit4]    > Assumption #1: The MockRandom PF randomizes content on the fly, so we can't check it
   [junit4] Completed [147/486] on J0 in 30.75s, 25 tests, 2 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestMatchNoDocsQuery
   [junit4] Completed [148/486] on J0 in 0.08s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSortedSetSelector
   [junit4] Completed [149/486] on J0 in 0.36s, 15 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterCommit
   [junit4] Completed [150/486] on J2 in 2.61s, 14 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestSegmentInfos
   [junit4] Completed [151/486] on J2 in 0.05s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestFieldCacheRewriteMethod
   [junit4] Completed [152/486] on J2 in 4.91s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestSimpleFSLockFactory
   [junit4] Completed [153/486] on J0 in 8.25s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestCustomTermFreq
   [junit4] Completed [154/486] on J0 in 0.13s, 12 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestFieldInfos
   [junit4] Completed [155/486] on J0 in 0.08s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPostingsOffsets
   [junit4] Completed [156/486] on J0 in 1.44s, 13 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestPagedBytes
   [junit4] Completed [157/486] on J0 in 4.47s, 4 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestUniqueTermCount
   [junit4] Completed [158/486] on J0 in 0.28s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestRollingBuffer
   [junit4] Completed [159/486] on J0 in 0.74s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPointValues
   [junit4] Completed [160/486] on J2 in 10.46s, 32 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestForceMergeForever
   [junit4] Completed [161/486] on J0 in 2.05s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestPersistentSnapshotDeletionPolicy
   [junit4] Completed [162/486] on J0 in 2.27s, 13 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestFilterDirectoryReader
   [junit4] Completed [163/486] on J0 in 0.08s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene50.TestLucene50TermVectorsFormat
   [junit4] Completed [164/486] on J1 in 31.82s, 17 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene50.TestLucene50StoredFieldsFormatHighCompression
   [junit4] IGNOR/A 0.00s J0 | TestLucene50StoredFieldsFormatHighCompression.testBigDocuments
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed [165/486] on J0 in 9.97s, 19 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTotalHitCountCollector
   [junit4] Completed [166/486] on J0 in 0.04s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestMultiLevelSkipList
   [junit4] Completed [167/486] on J0 in 0.17s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestPositionIncrement
   [junit4] Completed [168/486] on J0 in 0.06s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.fst.TestFSTs
   [junit4] IGNOR/A 0.00s J1 | TestFSTs.testBigSet
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed [169/486] on J1 in 12.65s, 20 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.Test4GBStoredFields
   [junit4] IGNOR/A 0.00s J1 | Test4GBStoredFields.test
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed [170/486] on J1 in 0.05s, 1 test, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestNumericUtils
   [junit4] Completed [171/486] on J1 in 2.77s, 24 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSortRescorer
   [junit4] Completed [172/486] on J1 in 0.82s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestPrefixInBooleanQuery
   [junit4] Completed [173/486] on J1 in 0.82s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestNativeFSLockFactory
   [junit4] Completed [174/486] on J0 in 10.26s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.bkd.TestMutablePointsReaderUtils
   [junit4] Completed [175/486] on J1 in 2.11s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestTopFieldCollectorEarlyTermination
   [junit4] Completed [176/486] on J1 in 20.40s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestBytesRef
   [junit4] Completed [177/486] on J1 in 0.03s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestWildcard
   [junit4] Completed [178/486] on J1 in 0.22s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene70.TestLucene70NormsFormat
   [junit4] IGNOR/A 0.00s J0 | TestLucene70NormsFormat.testSparseNCommonBig
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] IGNOR/A 0.00s J0 | TestLucene70NormsFormat.testMergeStability
   [junit4]    > Assumption #1: The MockRandom PF randomizes content on the fly, so we can't check it
   [junit4] IGNOR/A 0.00s J0 | TestLucene70NormsFormat.testNCommonBig
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed [179/486] on J0 in 24.57s, 29 tests, 3 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNoDeletionPolicy
   [junit4] Completed [180/486] on J0 in 0.54s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSimpleSearchEquivalence
   [junit4] Completed [181/486] on J0 in 1.97s, 17 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestConstantScoreQuery
   [junit4] Completed [182/486] on J0 in 0.08s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestCodecs
   [junit4] Completed [183/486] on J0 in 0.74s, 3 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.perfield.TestPerFieldDocValuesFormat
   [junit4] Completed [184/486] on J2 in 65.15s, 110 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterReader
   [junit4] IGNOR/A 0.00s J2 | TestIndexWriterReader.testDuringAddIndexes
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed [185/486] on J2 in 8.17s, 23 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.similarities.TestAxiomaticF3LOG
   [junit4] Completed [186/486] on J0 in 21.21s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging
   [junit4] Completed [187/486] on J2 in 5.56s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.store.TestGrowableByteArrayDataOutput
   [junit4] Completed [188/486] on J0 in 8.44s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestFilterIterator
   [junit4] Completed [189/486] on J0 in 0.04s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.automaton.LimitedFiniteStringsIteratorTest
   [junit4] Completed [190/486] on J0 in 0.34s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIntBlockPool
   [junit4] Completed [191/486] on J0 in 0.78s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestCharArrayMap
   [junit4] Completed [192/486] on J0 in 0.46s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestDocIdSetBuilder
   [junit4] Completed [193/486] on J0 in 2.73s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.packed.TestPackedInts
   [junit4] IGNOR/A 0.00s J2 | TestPackedInts.testBlockReaderOverflow
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed [194/486] on J2 in 9.71s, 28 tests, 3 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterExceptions
   [junit4] IGNOR/A 0.00s J2 | TestIndexWriterExceptions.testTooManyTokens
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed [195/486] on J2 in 9.34s, 34 tests, 1 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestBagOfPositions
   [junit4] Completed [196/486] on J0 in 12.34s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestMSBRadixSorter
   [junit4] Completed [197/486] on J0 in 3.43s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestNotDocIdSet
   [junit4] Completed [198/486] on J0 in 1.56s, 6 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestIOUtils
   [junit4] Completed [199/486] on J0 in 0.16s, 18 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexingSequenceNumbers
   [junit4] Completed [200/486] on J1 in 58.27s, 8 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestByteSlices
   [junit4] Completed [201/486] on J1 in 5.22s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene70.TestIndexedDISI
   [junit4] Completed [202/486] on J1 in 3.99s, 10 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.document.TestField
   [junit4] Completed [203/486] on J1 in 0.08s, 23 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSearcherManager
   [junit4] Completed [204/486] on J2 in 15.03s, 11 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTransactionRollback
   [junit4] Completed [205/486] on J1 in 0.12s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.TestCodecUtil
   [junit4] Completed [206/486] on J1 in 0.03s, 16 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestBooleanOr
   [junit4] Completed [207/486] on J2 in 1.71s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.spans.TestBasics
   [junit4] Completed [208/486] on J2 in 3.90s, 27 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestGraphTokenizers
   [junit4] Completed [209/486] on J1 in 6.80s, 23 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestNot
   [junit4] Completed [210/486] on J1 in 0.07s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestCharFilter
   [junit4] Completed [211/486] on J1 in 0.04s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestFlushByRamOrCountsPolicy
   [junit4] Completed [212/486] on J2 in 3.64s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestPhraseQuery
   [junit4] Completed [213/486] on J1 in 2.52s, 22 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestSloppyPhraseQuery
   [junit4] Completed [214/486] on J2 in 1.37s, 9 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.codecs.lucene50.TestLucene50CompoundFormat
   [junit4] Completed [215/486] on J1 in 5.64s, 26 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestMultiDocValues
   [junit4] Completed [216/486] on J1 in 0.50s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestTerm
   [junit4] Completed [217/486] on J1 in 0.05s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterDelete
   [junit4] IGNOR/A 0.00s J2 | TestIndexWriterDelete.testApplyDeletesOnFlush
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] Completed [218/486] on J2 in 12.40s, 25 tests, 2 skipped
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterMergePolicy
   [junit4] Completed [219/486] on J2 in 1.69s, 7 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.similarities.TestAxiomaticF2LOG
   [junit4] Completed [220/486] on J1 in 15.53s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.analysis.TestReusableStringReader
   [junit4] Completed [221/486] on J1 in 0.02s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestOmitTf
   [junit4] Completed [222/486] on J1 in 0.24s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.util.TestIntArrayDocIdSet
   [junit4] Completed [223/486] on J1 in 1.04s, 5 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestConcurrentMergeScheduler
   [junit4] Completed [224/486] on J1 in 5.28s, 16 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNewestSegment
   [junit4] Completed [225/486] on J1 in 0.07s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestDuelingCodecs
   [junit4] Completed [226/486] on J2 in 13.11s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TermInSetQueryTest
   [junit4] Completed [227/486] on J1 in 1.60s, 10 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestIsCurrent
   [junit4] Completed [228/486] on J1 in 0.06s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.index.TestNorms
   [junit4] Completed [229/486] on J1 in 1.55s, 2 tests
   [junit4] 
   [junit4] Suite: org.apache.lucene.search.TestBoolean2
   [junit4] Completed [230/486] on J2 in 4.49s, 10 te

[...truncated too long message...]

ttings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/null1176666787
     [copy] Copying 40 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/null1176666787
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/null1176666787

resolve-example:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

resolve-server:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/null915672586
     [copy] Copying 238 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/null915672586
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/null915672586

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] 	confs: [default]
[ivy:cachepath] 	found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] 	found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] 	found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] 	found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] 	found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] 	found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] 	found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] 	found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 159ms :: artifacts dl 11ms
	---------------------------------------------------------------------
	|                  |            modules            ||   artifacts   |
	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
	---------------------------------------------------------------------
	|      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
	---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 108 minutes 2 seconds
Archiving artifacts
WARN: No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
WARN: java.lang.InterruptedException: no matches found within 10000
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 6 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/6/

7 tests failed.
FAILED:  org.apache.solr.cloud.ConcurrentCreateRoutedAliasTest.testConcurrentCreateRoutedAliasComplex

Error Message:
concurrent alias creation failed org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:57481/solr: Can't create collection alias for collections='testAliasCplx0_2017-12-25_23_24_25', 'testAliasCplx0_2017-12-25_23_24_25' is not an existing collection or alias

Stack Trace:
java.lang.AssertionError: concurrent alias creation failed org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:57481/solr: Can't create collection alias for collections='testAliasCplx0_2017-12-25_23_24_25', 'testAliasCplx0_2017-12-25_23_24_25' is not an existing collection or alias
	at __randomizedtesting.SeedInfo.seed([2BA7C7AEF14E474F:41F5BBE7B31084D4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.cloud.ConcurrentCreateRoutedAliasTest.testConcurrentCreateRoutedAliasComplex(ConcurrentCreateRoutedAliasTest.java:133)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([2BA7C7AEF14E474F:816A145C469D929F]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertFalse(Assert.java:68)
	at org.junit.Assert.assertFalse(Assert.java:79)
	at org.apache.solr.cloud.MoveReplicaTest.testFailedMove(MoveReplicaTest.java:303)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.request.TestV1toV2ApiMapper.testCommands

Error Message:
null value is not valid for name

Stack Trace:
java.lang.RuntimeException: null value is not valid for name
	at __randomizedtesting.SeedInfo.seed([1F12AC59F109861F:3E2E65A19B33667B]:0)
	at org.apache.solr.client.solrj.request.V1toV2ApiMapper$ActionInfo.lambda$null$2(V1toV2ApiMapper.java:108)
	at org.apache.solr.common.MapWriter.toMap(MapWriter.java:42)
	at org.apache.solr.common.MapWriter$1.put(MapWriter.java:45)
	at org.apache.solr.client.solrj.request.V1toV2ApiMapper$ActionInfo.lambda$serializeToV2Format$3(V1toV2ApiMapper.java:98)
	at org.apache.solr.common.MapWriter.toMap(MapWriter.java:42)
	at org.apache.solr.client.solrj.request.V1toV2ApiMapper$ActionInfo.convert(V1toV2ApiMapper.java:87)
	at org.apache.solr.client.solrj.request.V1toV2ApiMapper.convert(V1toV2ApiMapper.java:137)
	at org.apache.solr.client.solrj.request.TestV1toV2ApiMapper.testCommands(TestV1toV2ApiMapper.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN

Error Message:
expected:<1.0> but was:<0.0>

Stack Trace:
java.lang.AssertionError: expected:<1.0> but was:<0.0>
	at __randomizedtesting.SeedInfo.seed([5B92C4E830354075:AA33B6B8058E8AE7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:443)
	at org.junit.Assert.assertEquals(Assert.java:512)
	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ZkControllerTest.testPublishAndWaitForDownStates

Error Message:
The ZkController.publishAndWaitForDownStates should have timed out but it didn't

Stack Trace:
java.lang.AssertionError: The ZkController.publishAndWaitForDownStates should have timed out but it didn't
	at __randomizedtesting.SeedInfo.seed([2BA7C7AEF14E474F:CA8AB4250D90870]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.ZkControllerTest.testPublishAndWaitForDownStates(ZkControllerTest.java:306)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload

Error Message:
Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>

Stack Trace:
java.lang.AssertionError: Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>
	at __randomizedtesting.SeedInfo.seed([2BA7C7AEF14E474F:291E1CC2820D9B82]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload(TestJmxIntegration.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart

Error Message:
IOException occured when talking to server at: http://127.0.0.1:39523/solr/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:39523/solr/collection1
	at __randomizedtesting.SeedInfo.seed([2BA7C7AEF14E474F:F350034A5A958513]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
	at org.apache.solr.handler.TestReplicationHandler.index(TestReplicationHandler.java:180)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart(TestReplicationHandler.java:643)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:39523 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 47 more




Build Log:
[...truncated 12105 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/init-core-data-001
   [junit4]   2> 57950 WARN  (SUITE-TestReplicationHandler-seed#[2BA7C7AEF14E474F]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=9 numCloses=9
   [junit4]   2> 57950 INFO  (SUITE-TestReplicationHandler-seed#[2BA7C7AEF14E474F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 57951 INFO  (SUITE-TestReplicationHandler-seed#[2BA7C7AEF14E474F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 57952 INFO  (SUITE-TestReplicationHandler-seed#[2BA7C7AEF14E474F]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 57955 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestReplicateAfterStartupWithNoActivity
   [junit4]   2> 57955 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/collection1
   [junit4]   2> 57959 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T15:27:37-06:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 57960 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 57961 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 57961 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 57961 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@69582d27{/solr,null,AVAILABLE}
   [junit4]   2> 57963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3bcacd16{HTTP/1.1,[http/1.1]}{127.0.0.1:41401}
   [junit4]   2> 57963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.Server Started @61077ms
   [junit4]   2> 57963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=41401}
   [junit4]   2> 57963 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 57963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 57963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 57963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 57963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-06T04:10:45.584Z
   [junit4]   2> 57964 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/solr.xml
   [junit4]   2> 57969 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 57969 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 57971 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 58056 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696
   [junit4]   2> 58101 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696
   [junit4]   2> 58101 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696
   [junit4]   2> 58105 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/.
   [junit4]   2> 58105 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 58119 INFO  (coreLoadExecutor-480-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 58143 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 58158 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 58163 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 58163 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 58164 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696
   [junit4]   2> 58164 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 58164 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 58182 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=49.6220703125, floorSegmentMB=1.0615234375, forceMergeDeletesPctAllowed=13.33158749406246, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 58346 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 58346 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 58347 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 58348 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@416d981a[collection1] main]
   [junit4]   2> 58348 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/collection1/conf
   [junit4]   2> 58349 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 58349 INFO  (coreLoadExecutor-480-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 58362 INFO  (searcherExecutor-481-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@416d981a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 58871 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-002/collection1
   [junit4]   2> 58875 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T15:27:37-06:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 58927 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 58927 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 58927 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 58961 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@d235af7{/solr,null,AVAILABLE}
   [junit4]   2> 58962 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@51e19d3d{HTTP/1.1,[http/1.1]}{127.0.0.1:60168}
   [junit4]   2> 58962 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.Server Started @62076ms
   [junit4]   2> 58962 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=60168}
   [junit4]   2> 58963 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 58963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 58963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 58963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 58963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-06T04:10:46.584Z
   [junit4]   2> 58963 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-002/solr.xml
   [junit4]   2> 58969 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 58969 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 58970 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 59033 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696
   [junit4]   2> 59047 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696
   [junit4]   2> 59048 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696
   [junit4]   2> 59051 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-002/.
   [junit4]   2> 59051 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 59052 INFO  (coreLoadExecutor-490-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 59095 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 59123 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 59126 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 59126 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 59126 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696
   [junit4]   2> 59127 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 59127 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-002/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 59137 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=49.6220703125, floorSegmentMB=1.0615234375, forceMergeDeletesPctAllowed=13.33158749406246, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 59224 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 59224 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 59226 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 59226 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@510c3d9f[collection1] main]
   [junit4]   2> 59227 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-002/collection1/conf
   [junit4]   2> 59242 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 59242 INFO  (coreLoadExecutor-490-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 59244 INFO  (searcherExecutor-491-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@510c3d9f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 59249 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@51e19d3d{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 59250 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1938588612
   [junit4]   2> 59250 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 59250 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6cc67f59: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@1a967d18
   [junit4]   2> 59258 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 59258 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5b8afdfd: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@4ad0b951
   [junit4]   2> 59273 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 59273 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@18f6ec07: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@5e7ef0e
   [junit4]   2> 59288 INFO  (coreCloseExecutor-496-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5e6cbc07
   [junit4]   2> 59288 INFO  (coreCloseExecutor-496-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1584184327
   [junit4]   2> 59288 INFO  (coreCloseExecutor-496-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7d0a581e: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@3db92849
   [junit4]   2> 59323 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@d235af7{/solr,null,UNAVAILABLE}
   [junit4]   2> 59323 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 59333 INFO  (qtp710899099-553) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 59335 INFO  (qtp710899099-553) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 3
   [junit4]   2> 59337 INFO  (qtp710899099-553) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 59337 INFO  (qtp710899099-553) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@5eb4a9a9 commitCommandVersion:0
   [junit4]   2> 59339 INFO  (qtp710899099-553) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4519460d[collection1] main]
   [junit4]   2> 59339 INFO  (qtp710899099-553) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 59341 INFO  (searcherExecutor-481-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4519460d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 59348 INFO  (qtp710899099-553) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 11
   [junit4]   2> 59354 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3bcacd16{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 59355 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=323478946
   [junit4]   2> 59355 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 59355 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4fc936d1: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@7eacd90b
   [junit4]   2> 59357 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 59357 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@539a59c3: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@4ad0b951
   [junit4]   2> 59358 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 59358 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6e62e2ed: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@5e7ef0e
   [junit4]   2> 59371 INFO  (coreCloseExecutor-498-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6bf1e09f
   [junit4]   2> 59371 INFO  (coreCloseExecutor-498-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1811013791
   [junit4]   2> 59371 INFO  (coreCloseExecutor-498-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1704cc0c: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@332b6415
   [junit4]   2> 59389 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@69582d27{/solr,null,UNAVAILABLE}
   [junit4]   2> 59389 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 59392 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T15:27:37-06:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 59395 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 59395 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 59395 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 59395 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@24f46080{/solr,null,AVAILABLE}
   [junit4]   2> 59396 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6640ff8b{HTTP/1.1,[http/1.1]}{127.0.0.1:53852}
   [junit4]   2> 59396 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.e.j.s.Server Started @62510ms
   [junit4]   2> 59396 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=53852}
   [junit4]   2> 59396 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 59397 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 59397 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 59397 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 59397 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-06T04:10:47.018Z
   [junit4]   2> 59397 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/solr.xml
   [junit4]   2> 59403 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 59403 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 59404 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 59469 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696
   [junit4]   2> 59482 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696
   [junit4]   2> 59482 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696
   [junit4]   2> 59485 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/.
   [junit4]   2> 59485 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity-seed#[2BA7C7AEF14E474F]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 59487 INFO  (coreLoadExecutor-505-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 59519 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 59529 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 59532 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 59532 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 59533 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75fd5696
   [junit4]   2> 59540 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 59540 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 59543 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=37, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.31124537200513636]
   [junit4]   2> 59606 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 59607 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 59608 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.49697676053392126]
   [junit4]   2> 59609 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6c8e21a6[collection1] main]
   [junit4]   2> 59609 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_2BA7C7AEF14E474F-001/solr-instance-001/collection1/conf
   [junit4]   2> 59610 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml
   [junit4]   2> 59610 INFO  (coreLoadExecutor-505-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 59612 INFO  (searcherExecutor-506-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6c8e21a6[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 59620 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 5
   [junit4]   2> 59623 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 0
   [junit4]   2> 59624 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 59626 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[3]} 0 0
   [junit4]   2> 59627 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[4]} 0 0
   [junit4]   2> 59628 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[5]} 0 0
   [junit4]   2> 59630 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[6]} 0 0
   [junit4]   2> 59631 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[7]} 0 0
   [junit4]   2> 59633 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[8]} 0 0
   [junit4]   2> 59634 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9]} 0 0
   [junit4]   2> 59635 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[10]} 0 0
   [junit4]   2> 59637 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[11]} 0 0
   [junit4]   2> 59638 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[12]} 0 0
   [junit4]   2> 59639 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[13]} 0 0
   [junit4]   2> 59641 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[14]} 0 0
   [junit4]   2> 59642 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[15]} 0 0
   [junit4]   2> 59643 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[16]} 0 0
   [junit4]   2> 59644 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[17]} 0 0
   [junit4]   2> 59646 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[18]} 0 0
   [junit4]   2> 59647 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[19]} 0 0
   [junit4]   2> 59648 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[20]} 0 0
   [junit4]   2> 59649 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[21]} 0 0
   [junit4]   2> 59651 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[22]} 0 0
   [junit4]   2> 59652 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[23]} 0 0
   [junit4]   2> 59653 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[24]} 0 0
   [junit4]   2> 59655 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[25]} 0 0
   [junit4]   2> 59661 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[26]} 0 5
   [junit4]   2> 59663 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[27]} 0 0
   [junit4]   2> 59664 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[28]} 0 0
   [junit4]   2> 59666 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[29]} 0 0
   [junit4]   2> 59674 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[30]} 0 0
   [junit4]   2> 59676 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[31]} 0 0
   [junit4]   2> 59678 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[32]} 0 0
   [junit4]   2> 59679 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[33]} 0 0
   [junit4]   2> 59681 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[34]} 0 0
   [junit4]   2> 59683 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[35]} 0 0
   [junit4]   2> 59685 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[36]} 0 0
   [junit4]   2> 59687 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[37]} 0 0
   [junit4]   2> 59689 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[38]} 0 0
   [junit4]   2> 59693 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[39]} 0 0
   [junit4]   2> 59703 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[40]} 0 0
   [junit4]   2> 59704 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[41]} 0 0
   [junit4]   2> 59706 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[42]} 0 1
   [junit4]   2> 59714 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[43]} 0 0
   [junit4]   2> 59716 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[44]} 0 0
   [junit4]   2> 59717 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[45]} 0 0
   [junit4]   2> 59723 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[46]} 0 0
   [junit4]   2> 59734 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[47]} 0 0
   [junit4]   2> 59735 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[48]} 0 0
   [junit4]   2> 59740 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[49]} 0 4
   [junit4]   2> 59742 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[50]} 0 0
   [junit4]   2> 59743 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[51]} 0 0
   [junit4]   2> 59744 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[52]} 0 0
   [junit4]   2> 59746 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[53]} 0 0
   [junit4]   2> 59747 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[54]} 0 0
   [junit4]   2> 59748 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[55]} 0 0
   [junit4]   2> 59749 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[56]} 0 0
   [junit4]   2> 59767 INFO  (qtp1344332938-595) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[57]} 0 0
   [junit4]   2> 59780 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[58]} 0 11
   [junit4]   2> 59787 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[59]} 0 0
   [junit4]   2> 59789 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[60]} 0 0
   [junit4]   2> 59790 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[61]} 0 0
   [junit4]   2> 59792 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[62]} 0 0
   [junit4]   2> 59793 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[63]} 0 0
   [junit4]   2> 59794 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[64]} 0 0
   [junit4]   2> 59796 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[65]} 0 0
   [junit4]   2> 59799 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[66]} 0 0
   [junit4]   2> 59801 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[67]} 0 0
   [junit4]   2> 59802 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[68]} 0 0
   [junit4]   2> 59803 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[69]} 0 0
   [junit4]   2> 59805 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[70]} 0 0
   [junit4]   2> 59806 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[71]} 0 0
   [junit4]   2> 59807 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[72]} 0 0
   [junit4]   2> 59809 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[73]} 0 0
   [junit4]   2> 59810 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[74]} 0 0
   [junit4]   2> 59811 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[75]} 0 0
   [junit4]   2> 59813 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[76]} 0 0
   [junit4]   2> 59814 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[77]} 0 0
   [junit4]   2> 59815 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[78]} 0 0
   [junit4]   2> 59817 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[79]} 0 0
   [junit4]   2> 59821 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[80]} 0 0
   [junit4]   2> 59822 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[81]} 0 0
   [junit4]   2> 59827 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[82]} 0 0
   [junit4]   2> 59829 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[83]} 0 0
   [junit4]   2> 59830 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[84]} 0 0
   [junit4]   2> 59832 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[85]} 0 0
   [junit4]   2> 59834 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[86]} 0 0
   [junit4]   2> 59836 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[87]} 0 0
   [junit4]   2> 59837 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[88]} 0 0
   [junit4]   2> 59839 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[89]} 0 0
   [junit4]   2> 59841 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[90]} 0 0
   [junit4]   2> 59845 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[91]} 0 0
   [junit4]   2> 59847 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[92]} 0 0
   [junit4]   2> 59848 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[93]} 0 0
   [junit4]   2> 59850 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[94]} 0 0
   [junit4]   2> 59851 INFO  (qtp1344332938-599) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[95]} 0 0
   [junit4]   2> 59852 INFO  (qtp1344332938-597) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection

[...truncated too long message...]

teErrorCases, TestCoreContainer, AtomicUpdateProcessorFactoryTest, TestCollationFieldDocValues, DistanceFunctionTest, TestRuleBasedAuthorizationPlugin, TestSystemCollAutoCreate, CollectionTooManyReplicasTest, SpatialHeatmapFacetsTest, OverseerModifyCollectionTest, DocValuesNotIndexedTest, TestStressReorder, TestPayloadScoreQParserPlugin, ShowFileRequestHandlerTest, SearchHandlerTest, TestConfigOverlay, TestJsonRequest, SpellCheckCollatorWithCollapseTest, PreAnalyzedUpdateProcessorTest, ConcurrentDeleteAndCreateCollectionTest, SearchHandlerTest, DateMathParserTest, ClassificationUpdateProcessorIntegrationTest, SolrCLIZkUtilsTest, TestAnalyzedSuggestions, TestClusterProperties, ClusterStateTest, RemoteQueryErrorTest, TestLegacyTerms, HdfsDirectoryFactoryTest, InfixSuggestersTest, TestLFUCache, SpellCheckComponentTest, NumericFieldsTest, CdcrUpdateLogTest, CoreAdminHandlerTest, UpdateLogTest, TestPerFieldSimilarityWithDefaultOverride, TestRealTimeGet, TestUninvertingReader, AutoAddReplicasPlanActionTest, TestReloadDeadlock, TestSolrConfigHandler, TestJavabinTupleStreamParser, ExecutePlanActionTest, TestRestManager, TestBulkSchemaConcurrent, DistributedDebugComponentTest, ZkShardTermsTest, TestNoOpRegenerator, TestMaxScoreQueryParser, TestDistributedStatsComponentCardinality, SolrShardReporterTest, TestBadConfig, MissingSegmentRecoveryTest, StatsReloadRaceTest, RecoveryZkTest, CreateRoutedAliasTest, ScheduledTriggerTest, SolrMetricManagerTest, TestCloudPseudoReturnFields, UniqFieldsUpdateProcessorFactoryTest, CoreMergeIndexesAdminHandlerTest, TestSmileRequest, LeaderElectionTest, TestFastWriter, URLClassifyProcessorTest, TestFileDictionaryLookup, TestNodeLostTrigger, TestConfigSetsAPIZkFailure, TestSolrDeletionPolicy2, UUIDUpdateProcessorFallbackTest, AutoscalingHistoryHandlerTest, MinimalSchemaTest, UninvertDocValuesMergePolicyTest, MoveReplicaTest, TestDFRSimilarityFactory, TestStressLiveNodes, TestComputePlanAction, TestGenericDistributedQueue, FileBasedSpellCheckerTest, TestFiltering, TestTolerantUpdateProcessorRandomCloud, SuggesterTSTTest, ConfigSetsAPITest, TestReplicaProperties, TestQueryTypes, TestRandomDVFaceting, TestRangeQuery, PropertiesRequestHandlerTest, TestMaxTokenLenTokenizer, CopyFieldTest, MoreLikeThisHandlerTest, TestFieldCache, CurrencyFieldTypeTest, HdfsRecoveryZkTest, TestNestedDocsSort, MetricUtilsTest, LukeRequestHandlerTest, SpatialFilterTest, TestCollectionsAPIViaSolrCloudCluster, ResponseLogComponentTest, TestEmbeddedSolrServerConstructors, TestDistribIDF, SSLMigrationTest, TestLeaderInitiatedRecoveryThread, LeaderInitiatedRecoveryOnShardRestartTest, LargeFieldTest, HttpSolrCallGetCoreTest, TestTrieFacet, TestUtils, DocExpirationUpdateProcessorFactoryTest, TestDynamicFieldCollectionResource, CoreSorterTest, DistribJoinFromCollectionTest, TestConfigsApi, ConvertedLegacyTest, AliasIntegrationTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, OverseerTest, ShardRoutingTest, TestFaceting, TestHashPartitioner, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, ZkControllerTest, TestJoin, SimpleFacetsTest, SolrCmdDistributorTest, BadIndexSchemaTest, TestLazyCores, DocValuesTest, TestSurroundQueryParser, TestQueryUtils, TestReversedWildcardFilterFactory, TermsComponentTest, FastVectorHighlighterTest, LoggingHandlerTest, RegexBoostProcessorTest, TestCollationField, TestCSVResponseWriter, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, PingRequestHandlerTest, CSVRequestHandlerTest, HighlighterConfigTest, TestQuerySenderListener, UpdateParamsTest, TestSolrIndexConfig, TestQuerySenderNoQuery, BadComponentTest, MultiTermTest, TestConfig, OutputWriterTest, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCodecSupport, TestXIncludeConfig, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TimeZoneUtilsTest, ScriptEngineTest, TestFastLRUCache, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, DateFieldTest, DOMUtilTest, RAMDirectoryFactoryTest, TestLRUCache, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, CircularListTest, DistributedMLTComponentTest, CursorPagingTest, DistributedIntervalFacetingTest, SolrTestCaseJ4Test, TestDistributedMissingSort, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, TestTolerantSearch, TestLuceneIndexBackCompat, TestEmbeddedSolrServerSchemaAPI, TestJettySolrRunner, ConnectionReuseTest, AddReplicaTest, CleanupOldIndexTest, CollectionPropsTest, CollectionStateFormat2Test, CollectionsAPISolrJTest, LeaderElectionContextKeyTest, LeaderFailureAfterFreshStartTest, MoveReplicaHDFSTest]
   [junit4] Completed [555/781 (5!)] on J2 in 186.15s, 4 tests, 1 failure <<< FAILURES!

[...truncated 1681 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.request.TestV1toV2ApiMapper
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestV1toV2ApiMapper -Dtests.method=testCommands -Dtests.seed=1F12AC59F109861F -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=de-CH -Dtests.timezone=MST7MDT -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.02s J2 | TestV1toV2ApiMapper.testCommands <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: null value is not valid for name
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1F12AC59F109861F:3E2E65A19B33667B]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.request.V1toV2ApiMapper$ActionInfo.lambda$null$2(V1toV2ApiMapper.java:108)
   [junit4]    > 	at org.apache.solr.common.MapWriter.toMap(MapWriter.java:42)
   [junit4]    > 	at org.apache.solr.common.MapWriter$1.put(MapWriter.java:45)
   [junit4]    > 	at org.apache.solr.client.solrj.request.V1toV2ApiMapper$ActionInfo.lambda$serializeToV2Format$3(V1toV2ApiMapper.java:98)
   [junit4]    > 	at org.apache.solr.common.MapWriter.toMap(MapWriter.java:42)
   [junit4]    > 	at org.apache.solr.client.solrj.request.V1toV2ApiMapper$ActionInfo.convert(V1toV2ApiMapper.java:87)
   [junit4]    > 	at org.apache.solr.client.solrj.request.V1toV2ApiMapper.convert(V1toV2ApiMapper.java:137)
   [junit4]    > 	at org.apache.solr.client.solrj.request.TestV1toV2ApiMapper.testCommands(TestV1toV2ApiMapper.java:55)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1119, maxMBSortInHeap=6.932355406521821, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@89b0266), locale=de-CH, timezone=MST7MDT
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=247343136,total=395837440
   [junit4]   2> NOTE: All tests run in this JVM: [TestCollectionStateWatchers, TestCloudSolrClientConnections, SolrExampleJettyTest, TestSuggesterResponse, ArcCosineEvaluatorTest, SelectWithEvaluatorsTest, OperationsTest, ArrayEvaluatorTest, HttpSolrClientBuilderTest, JettyWebappTest, CosineEvaluatorTest, GetByIdTest, CloudSolrClientBuilderTest, UuidEvaluatorTest, NamedListTest, GreaterThanEqualToEvaluatorTest, TestBatchUpdate, TestJsonRecordReader, ExclusiveOrEvaluatorTest, SolrExampleBinaryTest, CloudSolrClientTest, AddEvaluatorTest, TangentEvaluatorTest, TestJavaBinCodec, TestCollectionAdminRequest, ArcTangentEvaluatorTest, CeilingEvaluatorTest, CoalesceEvaluatorTest, CorrelationEvaluatorTest, CubedRootEvaluatorTest, DivideEvaluatorTest, EmpiricalDistributionEvaluatorTest, EqualToEvaluatorTest, GreaterThanEvaluatorTest, HyperbolicSineEvaluatorTest, LengthEvaluatorTest, LessThanEqualToEvaluatorTest, ModuloEvaluatorTest, MultiplyEvaluatorTest, NaturalLogEvaluatorTest, NormalDistributionEvaluatorTest, NormalizeEvaluatorTest, NotEvaluatorTest, PowerEvaluatorTest, RawValueEvaluatorTest, RegressionEvaluatorTest, SineEvaluatorTest, SolrPingTest, TestV1toV2ApiMapper]
   [junit4] Completed [134/146 (1!)] on J2 in 0.03s, 1 test, 1 error <<< FAILURES!

[...truncated 6575 lines...]
   [junit4] Suite: org.apache.solr.ltr.TestLTRReRankingPipeline
   [junit4]   2> 19331 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[5B92C4E830354075]) [    ] o.a.s.l.TestLTRReRankingPipeline rerank 1 documents 
   [junit4]   2> 19333 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[5B92C4E830354075]) [    ] o.a.s.l.TestLTRReRankingPipeline doc 0 in pos 0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTRReRankingPipeline -Dtests.method=testDifferentTopN -Dtests.seed=5B92C4E830354075 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=pt -Dtests.timezone=Asia/Urumqi -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.12s J2 | TestLTRReRankingPipeline.testDifferentTopN <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1.0> but was:<0.0>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([5B92C4E830354075:AA33B6B8058E8AE7]:0)
   [junit4]    > 	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@8daabb1), locale=pt, timezone=Asia/Urumqi
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=233837592,total=327155712
   [junit4]   2> NOTE: All tests run in this JVM: [TestOriginalScoreScorer, TestUserTermScorerQuery, TestRerankBase, TestExternalFeatures, TestAdapterModel, TestModelManager, TestLTRQParserExplain, TestLTRQParserPlugin, TestLTRReRankingPipeline]
   [junit4] Completed [16/37 (1!)] on J2 in 0.22s, 3 tests, 1 failure <<< FAILURES!

[...truncated 41987 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 5 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/5/

4 tests failed.
FAILED:  org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN

Error Message:
expected:<1.0> but was:<0.0>

Stack Trace:
java.lang.AssertionError: expected:<1.0> but was:<0.0>
	at __randomizedtesting.SeedInfo.seed([C3F479E35220566C:32550BB3679B9CFE]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:443)
	at org.junit.Assert.assertEquals(Assert.java:512)
	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ZkControllerTest.testPublishAndWaitForDownStates

Error Message:
The ZkController.publishAndWaitForDownStates should have timed out but it didn't

Stack Trace:
java.lang.AssertionError: The ZkController.publishAndWaitForDownStates should have timed out but it didn't
	at __randomizedtesting.SeedInfo.seed([8078486235829B49:A777248E9415D476]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.ZkControllerTest.testPublishAndWaitForDownStates(ZkControllerTest.java:306)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload

Error Message:
Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>

Stack Trace:
java.lang.AssertionError: Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>
	at __randomizedtesting.SeedInfo.seed([8078486235829B49:82C1930E46C14784]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload(TestJmxIntegration.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart

Error Message:
IOException occured when talking to server at: http://127.0.0.1:55132/solr/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:55132/solr/collection1
	at __randomizedtesting.SeedInfo.seed([8078486235829B49:588F8C869E595915]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
	at org.apache.solr.handler.TestReplicationHandler.index(TestReplicationHandler.java:180)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart(TestReplicationHandler.java:643)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:55132 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 47 more




Build Log:
[...truncated 12386 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> 221291 INFO  (SUITE-TestReplicationHandler-seed#[8078486235829B49]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/init-core-data-001
   [junit4]   2> 221291 WARN  (SUITE-TestReplicationHandler-seed#[8078486235829B49]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=30 numCloses=30
   [junit4]   2> 221294 INFO  (SUITE-TestReplicationHandler-seed#[8078486235829B49]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 221295 INFO  (SUITE-TestReplicationHandler-seed#[8078486235829B49]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 221298 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestIndexFetchOnMasterRestart
   [junit4]   2> 221299 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/collection1
   [junit4]   2> 221305 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 221317 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 221317 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 221317 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 221317 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2d387ce9{/solr,null,AVAILABLE}
   [junit4]   2> 221318 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@26b1a399{HTTP/1.1,[http/1.1]}{127.0.0.1:55132}
   [junit4]   2> 221318 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.Server Started @224614ms
   [junit4]   2> 221318 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=55132}
   [junit4]   2> 221318 ERROR (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 221318 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 221318 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 221318 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 221319 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-05T04:50:59.130Z
   [junit4]   2> 221319 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/solr.xml
   [junit4]   2> 221331 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 221331 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 221350 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 221396 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 221409 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 221409 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 221412 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/.
   [junit4]   2> 221412 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 221412 INFO  (coreLoadExecutor-589-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 221436 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 221446 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 221450 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 221450 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 221451 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 221451 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 221451 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 221461 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2145231875, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 221515 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 221515 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 221516 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=25.53515625, floorSegmentMB=1.4501953125, forceMergeDeletesPctAllowed=2.850707588522434, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 221517 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@72fd312a[collection1] main]
   [junit4]   2> 221517 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/collection1/conf
   [junit4]   2> 221518 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 221518 INFO  (coreLoadExecutor-589-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 221521 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-002/collection1
   [junit4]   2> 221522 INFO  (searcherExecutor-590-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@72fd312a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 221527 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 221528 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 221529 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 221529 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 221529 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@290fb136{/solr,null,AVAILABLE}
   [junit4]   2> 221530 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3ef4d133{HTTP/1.1,[http/1.1]}{127.0.0.1:34354}
   [junit4]   2> 221530 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.Server Started @224825ms
   [junit4]   2> 221530 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=34354}
   [junit4]   2> 221530 ERROR (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 221530 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 221530 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 221530 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 221530 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-05T04:50:59.341Z
   [junit4]   2> 221531 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-002/solr.xml
   [junit4]   2> 221535 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 221535 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 221537 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 221583 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 221598 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 221598 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 221601 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-002/.
   [junit4]   2> 221601 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 221601 INFO  (coreLoadExecutor-599-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 221620 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 221630 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 221632 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 221632 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 221632 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 221632 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 221633 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-002/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 221635 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2145231875, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 221717 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 221717 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 221719 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=25.53515625, floorSegmentMB=1.4501953125, forceMergeDeletesPctAllowed=2.850707588522434, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 221719 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@46cdd44c[collection1] main]
   [junit4]   2> 221720 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-002/collection1/conf
   [junit4]   2> 221739 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 221739 INFO  (coreLoadExecutor-599-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 221741 INFO  (searcherExecutor-600-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@46cdd44c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 221744 INFO  (qtp1376169545-1259) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 221753 INFO  (qtp1376169545-1263) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 221754 INFO  (indexFetcher-604-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 221754 INFO  (indexFetcher-604-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 221754 INFO  (indexFetcher-604-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 221754 INFO  (indexFetcher-604-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 221754 INFO  (indexFetcher-604-thread-1) [    x:collection1] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
   [junit4]   2> 221757 INFO  (indexFetcher-604-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 221757 INFO  (indexFetcher-604-thread-1) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@4714c6f commitCommandVersion:0
   [junit4]   2> 221760 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@26b1a399{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 221760 INFO  (indexFetcher-604-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7adf095f[collection1] main]
   [junit4]   2> 221760 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1893957657
   [junit4]   2> 221760 INFO  (indexFetcher-604-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 221761 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 221761 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1aa9652: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@8b95792
   [junit4]   2> 221765 INFO  (searcherExecutor-600-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7adf095f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 221767 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 221767 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7919dae2: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@7ca983a8
   [junit4]   2> 221770 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 221770 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@731894af: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@225bd3f5
   [junit4]   2> 221788 INFO  (coreCloseExecutor-605-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5e35c447
   [junit4]   2> 221788 INFO  (coreCloseExecutor-605-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1580581959
   [junit4]   2> 221789 INFO  (coreCloseExecutor-605-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@36031b2e: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@74922028
   [junit4]   2> 221835 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2d387ce9{/solr,null,UNAVAILABLE}
   [junit4]   2> 221835 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 221836 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 221846 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 221846 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 221846 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 221847 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7315bcc2{/solr,null,AVAILABLE}
   [junit4]   2> 221847 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@45c83efa{HTTP/1.1,[http/1.1]}{127.0.0.1:55132}
   [junit4]   2> 221847 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.Server Started @225143ms
   [junit4]   2> 221847 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=55132}
   [junit4]   2> 221848 ERROR (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 221848 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 221848 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 221848 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 221848 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-05T04:50:59.659Z
   [junit4]   2> 221849 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/solr.xml
   [junit4]   2> 221853 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 221853 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 221854 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 222028 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 222046 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 222047 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 222060 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/.
   [junit4]   2> 222060 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 222061 INFO  (coreLoadExecutor-612-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 222089 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 222099 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 222101 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 222101 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 222101 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 222118 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 222118 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 222125 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=42, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=8.701171875, floorSegmentMB=1.8095703125, forceMergeDeletesPctAllowed=19.07167397011207, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 222315 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 222315 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 222317 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=31.935546875, floorSegmentMB=0.7099609375, forceMergeDeletesPctAllowed=18.76565992128332, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 222317 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5795b133[collection1] main]
   [junit4]   2> 222318 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-001/collection1/conf
   [junit4]   2> 222319 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml
   [junit4]   2> 222319 INFO  (coreLoadExecutor-612-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 222321 INFO  (searcherExecutor-613-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5795b133[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 222331 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.SolrTestCaseJ4 ###Ending doTestIndexFetchOnMasterRestart
   [junit4]   2> 222331 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@45c83efa{HTTP/1.1,[http/1.1]}{127.0.0.1:55132}
   [junit4]   2> 222332 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1032913585
   [junit4]   2> 222335 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 222336 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@67b6f033: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@5bd8bbd1
   [junit4]   2> 222343 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 222343 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1847c7f0: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@7ca983a8
   [junit4]   2> 222359 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 222359 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@a2eae28: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@225bd3f5
   [junit4]   2> 222361 INFO  (coreCloseExecutor-617-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3491c537
   [junit4]   2> 222361 INFO  (coreCloseExecutor-617-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=881968439
   [junit4]   2> 222361 INFO  (coreCloseExecutor-617-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@16287e12: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@1c16b0d0
   [junit4]   2> 222382 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7315bcc2{/solr,null,UNAVAILABLE}
   [junit4]   2> 222382 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 222383 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3ef4d133{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 222384 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=882853448
   [junit4]   2> 222385 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 222385 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@37f49b98: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@4810dd7a
   [junit4]   2> 222387 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 222387 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5ec1d4a1: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@7ca983a8
   [junit4]   2> 222388 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 222388 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@41e2195: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@225bd3f5
   [junit4]   2> 222404 INFO  (coreCloseExecutor-619-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@14769839
   [junit4]   2> 222404 INFO  (coreCloseExecutor-619-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=343316537
   [junit4]   2> 222404 INFO  (coreCloseExecutor-619-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2cfa41cb: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@2be6aa9
   [junit4]   2> 222419 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@290fb136{/solr,null,UNAVAILABLE}
   [junit4]   2> 222419 INFO  (TEST-TestReplicationHandler.doTestIndexFetchOnMasterRestart-seed#[8078486235829B49]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestReplicationHandler -Dtests.method=doTestIndexFetchOnMasterRestart -Dtests.seed=8078486235829B49 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=zh -Dtests.timezone=Etc/GMT-2 -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   1.14s J1 | TestReplicationHandler.doTestIndexFetchOnMasterRestart <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:55132/solr/collection1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8078486235829B49:588F8C869E595915]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
   [junit4]    > 	at org.apache.solr.handler.TestReplicationHandler.index(TestReplicationHandler.java:180)
   [junit4]    > 	at org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart(TestReplicationHandler.java:643)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:55132 failed to respond
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]    > 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]    > 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]    > 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]    > 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]    > 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]    > 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
   [junit4]    > 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]    > 	... 47 more
   [junit4]   2> 222432 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestStressReplication
   [junit4]   2> 222433 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-003/collection1
   [junit4]   2> 222436 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 222445 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 222445 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 222445 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 222448 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5d9f29ff{/solr,null,AVAILABLE}
   [junit4]   2> 222448 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@715f68dc{HTTP/1.1,[http/1.1]}{127.0.0.1:50615}
   [junit4]   2> 222449 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.Server Started @225744ms
   [junit4]   2> 222449 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-003/collection1/data, hostContext=/solr, hostPort=50615}
   [junit4]   2> 222449 ERROR (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 222449 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 222449 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 222449 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 222449 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-05T04:51:00.260Z
   [junit4]   2> 222450 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-003/solr.xml
   [junit4]   2> 222463 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 222463 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 222464 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 222519 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 222531 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 222534 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 222539 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-003/.
   [junit4]   2> 222539 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 222539 INFO  (coreLoadExecutor-626-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 222578 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 222594 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 222597 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 222597 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-003/./collection1, trusted=true
   [junit4]   2> 222597 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 222597 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 222597 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-003/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-003/./collection1/data/]
   [junit4]   2> 222600 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2145231875, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 222654 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 222654 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 222655 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=25.53515625, floorSegmentMB=1.4501953125, forceMergeDeletesPctAllowed=2.850707588522434, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 222656 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2b356630[collection1] main]
   [junit4]   2> 222656 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-003/collection1/conf
   [junit4]   2> 222657 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 222657 INFO  (coreLoadExecutor-626-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 222659 INFO  (searcherExecutor-627-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2b356630[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 222660 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-004/collection1
   [junit4]   2> 222663 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 222665 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 222665 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 222665 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 222665 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@29fbfd1{/solr,null,AVAILABLE}
   [junit4]   2> 222666 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@72ae4ff4{HTTP/1.1,[http/1.1]}{127.0.0.1:38605}
   [junit4]   2> 222666 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.Server Started @225962ms
   [junit4]   2> 222666 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-004/collection1/data, hostContext=/solr, hostPort=38605}
   [junit4]   2> 222666 ERROR (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 222666 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 222666 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 222666 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 222666 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-05T04:51:00.477Z
   [junit4]   2> 222667 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-004/solr.xml
   [junit4]   2> 222671 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 222671 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 222672 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 222719 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 222733 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 222734 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 222737 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-004/.
   [junit4]   2> 222737 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 222737 INFO  (coreLoadExecutor-636-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 222750 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 222759 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 222761 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 222761 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-004/./collection1, trusted=true
   [junit4]   2> 222762 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@61beb67d
   [junit4]   2> 222762 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 222762 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-004/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-004/./collection1/data/]
   [junit4]   2> 222764 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2145231875, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 222822 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 222822 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 222823 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=25.53515625, floorSegmentMB=1.4501953125, forceMergeDeletesPctAllowed=2.850707588522434, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 222823 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1dec07b5[collection1] main]
   [junit4]   2> 222824 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_8078486235829B49-001/solr-instance-004/collection1/conf
   [junit4]   2> 222826 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 222826 INFO  (coreLoadExecutor-636-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 222829 INFO  (searcherExecutor-637-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1dec07b5[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 222835 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@72ae4ff4{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 222835 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=332281923
   [junit4]   2> 222842 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 222842 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6d3e97f9: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@2b76e460
   [junit4]   2> 222842 INFO  (qtp2012883132-1322) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 222844 INFO  (indexFetcher-641-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 222845 INFO  (indexFetcher-641-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 222845 INFO  (indexFetcher-641-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 222845 INFO  (indexFetcher-641-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 222848 INFO  (indexFetcher-641-thread-1) [    x:collection1] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
   [junit4]   2> 222851 INFO  (indexFetcher-641-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 222851 INFO  (indexFetcher-641-thread-1) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@13271288 commitCommandVersion:0
   [junit4]   2> 222859 INFO  (indexFetcher-641-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@489ad76b[collection1] main]
   [junit4]   2> 222859 INFO  (indexFetcher-641-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 222861 INFO  (searcherExecutor-637-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@489ad76b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 222864 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 222864 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@f6e51fb: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@7ca983a8
   [junit4]   2> 222870 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 222870 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@418d288d: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@225bd3f5
   [junit4]   2> 222871 INFO  (coreCloseExecutor-642-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3b251aa3
   [junit4]   2> 222871 INFO  (coreCloseExecutor-642-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=992287395
   [junit4]   2> 222871 INFO  (coreCloseExecutor-642-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3072a9cc: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@404f795f
   [junit4]   2> 222890 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@29fbfd1{/solr,null,UNAVAILABLE}
   [junit4]   2> 222890 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 222892 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 222901 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 222901 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 222901 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 222902 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@57f0d287{/solr,null,AVAILABLE}
   [junit4]   2> 222902 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[8078486235829B49]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4030c57c{HTTP/1.1,[http/1.1]}{127.0.0.1:46531}
   [junit4]   2> 222903 INFO  (TEST-TestReplicationHandler.doTestStressReplication-see

[...truncated too long message...]

TestJmxIntegration.testJmxUpdate-seed#[8078486235829B49]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1594073406985207808,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1629565 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[8078486235829B49]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@7fc797c9 commitCommandVersion:1594073406985207808
   [junit4]   2> 1629624 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[8078486235829B49]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@11536066[collection1] main]
   [junit4]   2> 1629625 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[8078486235829B49]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1629628 INFO  (searcherExecutor-5374-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@11536066[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):c1)))}
   [junit4]   2> 1629633 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[8078486235829B49]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 68
   [junit4]   2> 1629634 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[8078486235829B49]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testJmxUpdate
   [junit4]   2> 1629654 INFO  (TEST-TestJmxIntegration.testJmxRegistration-seed#[8078486235829B49]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testJmxRegistration
   [junit4]   2> 1629867 INFO  (TEST-TestJmxIntegration.testJmxRegistration-seed#[8078486235829B49]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testJmxRegistration
   [junit4]   2> 1629867 INFO  (SUITE-TestJmxIntegration-seed#[8078486235829B49]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 1629867 INFO  (SUITE-TestJmxIntegration-seed#[8078486235829B49]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2079104951
   [junit4]   2> 1629868 INFO  (SUITE-TestJmxIntegration-seed#[8078486235829B49]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1629868 INFO  (SUITE-TestJmxIntegration-seed#[8078486235829B49]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3c6c27c3: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@31a2ffd9
   [junit4]   2> 1629879 INFO  (SUITE-TestJmxIntegration-seed#[8078486235829B49]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1629879 INFO  (SUITE-TestJmxIntegration-seed#[8078486235829B49]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4b062e5e: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@2ed62e3f
   [junit4]   2> 1629883 INFO  (SUITE-TestJmxIntegration-seed#[8078486235829B49]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1629883 INFO  (SUITE-TestJmxIntegration-seed#[8078486235829B49]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6bafbbc: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@5dfdd7c1
   [junit4]   2> 1629884 INFO  (coreCloseExecutor-5379-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@21cae71b
   [junit4]   2> 1629884 INFO  (coreCloseExecutor-5379-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=566945563
   [junit4]   2> 1629884 INFO  (coreCloseExecutor-5379-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@453e1340: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@1024c3c7
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.core.TestJmxIntegration_8078486235829B49-001
   [junit4]   2> NOTE: test params are: codec=SimpleText, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@574ce036), locale=de-DE, timezone=EST
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=178167376,total=530579456
   [junit4]   2> NOTE: All tests run in this JVM: [TestSubQueryTransformer, GraphQueryTest, RecoveryZkTest, DistribCursorPagingTest, TestSha256AuthenticationProvider, CachingDirectoryFactoryTest, TestRetrieveFieldsOptimizer, DistanceUnitsTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, CleanupOldIndexTest, ExplicitHLLTest, TestSolrXml, TestStressReorder, TestLargeCluster, CdcrReplicationDistributedZkTest, TestCorePropertiesReload, TestDistributedMissingSort, TestUtilizeNode, TestFreeTextSuggestions, TestCoreDiscovery, DataDrivenBlockJoinTest, TestCloudJSONFacetJoinDomain, TestAuthorizationFramework, TestSolr4Spatial2, TestRequestStatusCollectionAPI, CoreAdminCreateDiscoverTest, TestPolicyCloud, DistributedQueueTest, TimeZoneUtilsTest, TestPrepRecovery, SpellCheckComponentTest, TestTolerantSearch, TestCoreAdminApis, BlockDirectoryTest, TestRangeQuery, NotRequiredUniqueKeyTest, HdfsBasicDistributedZkTest, TestNumericTokenStream, LeaderFailoverAfterPartitionTest, ReplicaListTransformerTest, TestRandomFlRTGCloud, AutoScalingHandlerTest, TestDFISimilarityFactory, TestDFRSimilarityFactory, TestFieldTypeCollectionResource, TriggerIntegrationTest, LeaderElectionIntegrationTest, StatelessScriptUpdateProcessorFactoryTest, PingRequestHandlerTest, InfoHandlerTest, TestSuggestSpellingConverter, TestSolrConfigHandlerCloud, TestLazyCores, TestSolrCloudWithHadoopAuthPlugin, ResponseLogComponentTest, CollectionsAPIAsyncDistributedZkTest, TestExpandComponent, UnloadDistributedZkTest, ClusterStateTest, SuggesterTSTTest, TestJsonFacetRefinement, DeleteInactiveReplicaTest, TestAddFieldRealTimeGet, VersionInfoTest, TestObjectReleaseTracker, DirectSolrSpellCheckerTest, DistributedSpellCheckComponentTest, IndexSchemaRuntimeFieldTest, TestQuerySenderListener, DeleteReplicaTest, SolrCloudExampleTest, TestHdfsCloudBackupRestore, TestComplexPhraseQParserPlugin, TestMissingGroups, TestOverriddenPrefixQueryForCustomFieldType, WrapperMergePolicyFactoryTest, TestBinaryResponseWriter, CopyFieldTest, HdfsLockFactoryTest, AtomicUpdatesTest, TestTestInjection, FileBasedSpellCheckerTest, TestEmbeddedSolrServerConstructors, SolrCmdDistributorTest, TestTlogReplica, LukeRequestHandlerTest, TestStandardQParsers, BigEndianAscendingWordDeserializerTest, ConnectionReuseTest, ClassificationUpdateProcessorTest, OverseerRolesTest, ReplaceNodeNoTargetTest, TestSolrCoreProperties, TestFaceting, TestInPlaceUpdatesDistrib, TestLockTree, AnalysisAfterCoreReloadTest, CurrencyRangeFacetCloudTest, TestSerializedLuceneMatchVersion, TestLeaderElectionWithEmptyReplica, DateRangeFieldTest, QueryResultKeyTest, SearchHandlerTest, UniqFieldsUpdateProcessorFactoryTest, TestMacros, TestAnalyzeInfixSuggestions, HdfsNNFailoverTest, TestHalfAndHalfDocValues, HLLUtilTest, MetricsConfigTest, ZkStateWriterTest, RegexBytesRefFilterTest, AliasIntegrationTest, MergeStrategyTest, TestUseDocValuesAsStored2, TestRandomDVFaceting, TestDynamicFieldResource, V2ApiIntegrationTest, TestBadConfig, TestDistribStateManager, ChaosMonkeyNothingIsSafeWithPullReplicasTest, ClusterStateUpdateTest, OverriddenZkACLAndCredentialsProvidersTest, TestOrdValues, ResponseHeaderTest, NumericFieldsTest, TestJmxIntegration]
   [junit4] Completed [539/780 (3!)] on J2 in 1.19s, 3 tests, 1 failure <<< FAILURES!

[...truncated 8274 lines...]
   [junit4] Suite: org.apache.solr.ltr.TestLTRReRankingPipeline
   [junit4]   2> 25061 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[C3F479E35220566C]) [    ] o.a.s.l.TestLTRReRankingPipeline rerank 1 documents 
   [junit4]   2> 25062 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[C3F479E35220566C]) [    ] o.a.s.l.TestLTRReRankingPipeline doc 0 in pos 0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTRReRankingPipeline -Dtests.method=testDifferentTopN -Dtests.seed=C3F479E35220566C -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ja-JP-u-ca-japanese-x-lvariant-JP -Dtests.timezone=Africa/Niamey -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.02s J1 | TestLTRReRankingPipeline.testDifferentTopN <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1.0> but was:<0.0>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C3F479E35220566C:32550BB3679B9CFE]:0)
   [junit4]    > 	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2c26a6d2), locale=ja-JP-u-ca-japanese-x-lvariant-JP, timezone=Africa/Niamey
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=243225240,total=355991552
   [junit4]   2> NOTE: All tests run in this JVM: [TestUserTermScoreWithQ, TestValueFeature, TestFieldValueFeature, TestOriginalScoreFeature, TestEdisMaxSolrFeature, TestFeatureExtractionFromMultipleSegments, TestLTRReRankingPipeline]
   [junit4] Completed [15/37 (1!)] on J1 in 0.13s, 3 tests, 1 failure <<< FAILURES!

[...truncated 40365 lines...]

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 4 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/4/

6 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:10010_solr should have been deleted

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:10010_solr should have been deleted
	at __randomizedtesting.SeedInfo.seed([42B4BD78D69635DF:5A0E3574D8A3F830]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertFalse(Assert.java:68)
	at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration(TestTriggerIntegration.java:844)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.security.TestPKIAuthenticationPlugin.test

Error Message:


Stack Trace:
java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([42B4BD78D69635DF:CAE082A2786A5827]:0)
	at org.apache.solr.security.TestPKIAuthenticationPlugin.test(TestPKIAuthenticationPlugin.java:102)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN

Error Message:
expected:<1.0> but was:<0.0>

Stack Trace:
java.lang.AssertionError: expected:<1.0> but was:<0.0>
	at __randomizedtesting.SeedInfo.seed([1C72B4201CB89142:EDD3C67029035BD0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:443)
	at org.junit.Assert.assertEquals(Assert.java:512)
	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ZkControllerTest.testPublishAndWaitForDownStates

Error Message:
The ZkController.publishAndWaitForDownStates should have timed out but it didn't

Stack Trace:
java.lang.AssertionError: The ZkController.publishAndWaitForDownStates should have timed out but it didn't
	at __randomizedtesting.SeedInfo.seed([42B4BD78D69635DF:65BBD19477017AE0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.ZkControllerTest.testPublishAndWaitForDownStates(ZkControllerTest.java:306)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload

Error Message:
Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>

Stack Trace:
java.lang.AssertionError: Number of registered MBeans is not the same as info registry size expected:<107> but was:<0>
	at __randomizedtesting.SeedInfo.seed([42B4BD78D69635DF:400D6614A5D5E912]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload(TestJmxIntegration.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart

Error Message:
IOException occured when talking to server at: http://127.0.0.1:34959/solr/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:34959/solr/collection1
	at __randomizedtesting.SeedInfo.seed([42B4BD78D69635DF:9A43799C7D4DF783]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
	at org.apache.solr.handler.TestReplicationHandler.index(TestReplicationHandler.java:180)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexFetchOnMasterRestart(TestReplicationHandler.java:643)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:34959 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 47 more




Build Log:
[...truncated 12414 lines...]
   [junit4] Suite: org.apache.solr.security.TestPKIAuthenticationPlugin
   [junit4]   2> 1162279 INFO  (SUITE-TestPKIAuthenticationPlugin-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.security.TestPKIAuthenticationPlugin_42B4BD78D69635DF-001/init-core-data-001
   [junit4]   2> 1162280 WARN  (SUITE-TestPKIAuthenticationPlugin-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=10 numCloses=10
   [junit4]   2> 1162280 INFO  (SUITE-TestPKIAuthenticationPlugin-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1162282 INFO  (SUITE-TestPKIAuthenticationPlugin-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1162301 INFO  (TEST-TestPKIAuthenticationPlugin.test-seed#[42B4BD78D69635DF]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 1173835 ERROR (TEST-TestPKIAuthenticationPlugin.test-seed#[42B4BD78D69635DF]) [    ] o.a.s.s.PKIAuthenticationPlugin Invalid key request timestamp: 1520142768951 , received timestamp: 1520142780119 , TTL: 10000
   [junit4]   2> 1173868 INFO  (TEST-TestPKIAuthenticationPlugin.test-seed#[42B4BD78D69635DF]) [    ] o.a.s.SolrTestCaseJ4 ###Ending test
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestPKIAuthenticationPlugin -Dtests.method=test -Dtests.seed=42B4BD78D69635DF -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=es-MX -Dtests.timezone=Australia/Victoria -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   11.6s J2 | TestPKIAuthenticationPlugin.test <<<
   [junit4]    > Throwable #1: java.lang.NullPointerException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([42B4BD78D69635DF:CAE082A2786A5827]:0)
   [junit4]    > 	at org.apache.solr.security.TestPKIAuthenticationPlugin.test(TestPKIAuthenticationPlugin.java:102)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J2/temp/solr.security.TestPKIAuthenticationPlugin_42B4BD78D69635DF-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=69, maxMBSortInHeap=7.744719736758107, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@41ee5930), locale=es-MX, timezone=Australia/Victoria
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=277908864,total=516423680
   [junit4]   2> NOTE: All tests run in this JVM: [TestStressInPlaceUpdates, BasicDistributedZk2Test, TestTolerantSearch, TestSlowCompositeReaderWrapper, DateFieldTest, TestUniqueKeyFieldResource, TestPullReplica, LeaderFailoverAfterPartitionTest, TestBlendedInfixSuggestions, TestManagedSynonymGraphFilterFactory, TestCollectionAPIs, TestCoreDiscovery, TestLazyCores, PreAnalyzedFieldManagedSchemaCloudTest, CheckHdfsIndexTest, TestPKIAuthenticationPlugin]
   [junit4] Completed [106/779 (1!)] on J2 in 11.73s, 1 test, 1 error <<< FAILURES!

[...truncated 284 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> 1646367 INFO  (SUITE-TestReplicationHandler-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/init-core-data-001
   [junit4]   2> 1646367 WARN  (SUITE-TestReplicationHandler-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=12 numCloses=12
   [junit4]   2> 1646367 INFO  (SUITE-TestReplicationHandler-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1646369 INFO  (SUITE-TestReplicationHandler-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 1646425 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestIndexAndConfigReplication
   [junit4]   2> 1646425 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-001/collection1
   [junit4]   2> 1646452 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T02:27:37+05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1646476 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1646477 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1646477 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1646477 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1ccc719{/solr,null,AVAILABLE}
   [junit4]   2> 1646478 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@32eba378{HTTP/1.1,[http/1.1]}{127.0.0.1:45237}
   [junit4]   2> 1646478 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.e.j.s.Server Started @1658070ms
   [junit4]   2> 1646478 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=45237}
   [junit4]   2> 1646478 ERROR (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1646478 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1646478 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1646478 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1646478 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-04T06:00:52.445Z
   [junit4]   2> 1646495 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-001/solr.xml
   [junit4]   2> 1646499 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1646499 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1646512 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@57bbabd1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1646646 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@57bbabd1
   [junit4]   2> 1646676 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@57bbabd1
   [junit4]   2> 1646676 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@57bbabd1
   [junit4]   2> 1646678 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-001/.
   [junit4]   2> 1646678 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1646690 INFO  (coreLoadExecutor-2582-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1646782 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1646817 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1646819 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1646819 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 1646819 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@57bbabd1
   [junit4]   2> 1646832 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1646832 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-001/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1646835 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=94.0029296875, floorSegmentMB=1.9921875, forceMergeDeletesPctAllowed=5.512345952795357, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35057928942457417
   [junit4]   2> 1647068 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1647068 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1647069 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=10, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1647069 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7f09c3d8[collection1] main]
   [junit4]   2> 1647070 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-001/collection1/conf
   [junit4]   2> 1647070 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 1647070 INFO  (coreLoadExecutor-2582-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1647071 INFO  (searcherExecutor-2583-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7f09c3d8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1647091 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-002/collection1
   [junit4]   2> 1647106 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T02:27:37+05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1647107 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1647107 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1647107 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1647107 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7ddcfbe4{/solr,null,AVAILABLE}
   [junit4]   2> 1647108 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@55aecacb{HTTP/1.1,[http/1.1]}{127.0.0.1:44664}
   [junit4]   2> 1647108 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.e.j.s.Server Started @1658700ms
   [junit4]   2> 1647108 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=44664}
   [junit4]   2> 1647108 ERROR (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1647108 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1647108 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1647108 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1647108 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-04T06:00:53.075Z
   [junit4]   2> 1647125 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-002/solr.xml
   [junit4]   2> 1647144 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1647144 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1647145 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@57bbabd1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1647400 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@57bbabd1
   [junit4]   2> 1647452 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@57bbabd1
   [junit4]   2> 1647452 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@57bbabd1
   [junit4]   2> 1647471 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-002/.
   [junit4]   2> 1647471 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[42B4BD78D69635DF]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1647471 INFO  (coreLoadExecutor-2592-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1647687 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1647706 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1647740 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1647740 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 1647741 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@57bbabd1
   [junit4]   2> 1647741 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1647741 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-002/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 1647742 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=94.0029296875, floorSegmentMB=1.9921875, forceMergeDeletesPctAllowed=5.512345952795357, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35057928942457417
   [junit4]   2> 1648034 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1648034 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1648035 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=10, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1648035 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4aa18c91[collection1] main]
   [junit4]   2> 1648056 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_42B4BD78D69635DF-001/solr-instance-002/collection1/conf
   [junit4]   2> 1648058 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 1648058 INFO  (coreLoadExecutor-2592-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1648059 INFO  (searcherExecutor-2593-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4aa18c91[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1648089 INFO  (qtp1778390070-6914) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1648091 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 1
   [junit4]   2> 1648092 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 0
   [junit4]   2> 1648104 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 1648105 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[3]} 0 0
   [junit4]   2> 1648105 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[4]} 0 0
   [junit4]   2> 1648108 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[5]} 0 0
   [junit4]   2> 1648125 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[6]} 0 0
   [junit4]   2> 1648141 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[7]} 0 0
   [junit4]   2> 1648142 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[8]} 0 0
   [junit4]   2> 1648142 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9]} 0 0
   [junit4]   2> 1648143 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[10]} 0 0
   [junit4]   2> 1648143 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[11]} 0 0
   [junit4]   2> 1648144 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[12]} 0 0
   [junit4]   2> 1648144 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[13]} 0 0
   [junit4]   2> 1648144 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[14]} 0 0
   [junit4]   2> 1648145 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[15]} 0 0
   [junit4]   2> 1648180 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[16]} 0 0
   [junit4]   2> 1648202 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[17]} 0 0
   [junit4]   2> 1648203 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[18]} 0 0
   [junit4]   2> 1648203 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[19]} 0 0
   [junit4]   2> 1648203 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[20]} 0 0
   [junit4]   2> 1648204 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[21]} 0 0
   [junit4]   2> 1648204 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[22]} 0 0
   [junit4]   2> 1648205 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[23]} 0 0
   [junit4]   2> 1648205 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[24]} 0 0
   [junit4]   2> 1648205 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[25]} 0 0
   [junit4]   2> 1648206 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[26]} 0 0
   [junit4]   2> 1648206 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[27]} 0 0
   [junit4]   2> 1648223 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[28]} 0 0
   [junit4]   2> 1648223 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[29]} 0 0
   [junit4]   2> 1648224 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[30]} 0 0
   [junit4]   2> 1648224 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[31]} 0 0
   [junit4]   2> 1648224 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[32]} 0 0
   [junit4]   2> 1648225 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[33]} 0 0
   [junit4]   2> 1648225 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[34]} 0 0
   [junit4]   2> 1648225 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[35]} 0 0
   [junit4]   2> 1648226 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[36]} 0 0
   [junit4]   2> 1648226 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[37]} 0 0
   [junit4]   2> 1648239 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[38]} 0 12
   [junit4]   2> 1648239 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[39]} 0 0
   [junit4]   2> 1648240 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[40]} 0 0
   [junit4]   2> 1648240 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[41]} 0 0
   [junit4]   2> 1648240 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[42]} 0 0
   [junit4]   2> 1648241 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[43]} 0 0
   [junit4]   2> 1648241 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[44]} 0 0
   [junit4]   2> 1648241 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[45]} 0 0
   [junit4]   2> 1648242 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[46]} 0 0
   [junit4]   2> 1648242 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[47]} 0 0
   [junit4]   2> 1648242 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[48]} 0 0
   [junit4]   2> 1648243 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[49]} 0 0
   [junit4]   2> 1648259 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[50]} 0 0
   [junit4]   2> 1648260 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[51]} 0 0
   [junit4]   2> 1648260 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[52]} 0 0
   [junit4]   2> 1648260 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[53]} 0 0
   [junit4]   2> 1648261 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[54]} 0 0
   [junit4]   2> 1648261 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[55]} 0 0
   [junit4]   2> 1648275 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[56]} 0 0
   [junit4]   2> 1648276 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[57]} 0 0
   [junit4]   2> 1648276 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[58]} 0 0
   [junit4]   2> 1648276 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[59]} 0 0
   [junit4]   2> 1648277 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[60]} 0 0
   [junit4]   2> 1648277 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[61]} 0 0
   [junit4]   2> 1648277 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[62]} 0 0
   [junit4]   2> 1648278 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[63]} 0 0
   [junit4]   2> 1648278 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[64]} 0 0
   [junit4]   2> 1648278 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[65]} 0 0
   [junit4]   2> 1648279 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[66]} 0 0
   [junit4]   2> 1648279 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[67]} 0 0
   [junit4]   2> 1648312 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[68]} 0 0
   [junit4]   2> 1648312 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[69]} 0 0
   [junit4]   2> 1648313 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[70]} 0 0
   [junit4]   2> 1648313 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[71]} 0 0
   [junit4]   2> 1648313 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[72]} 0 0
   [junit4]   2> 1648314 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[73]} 0 0
   [junit4]   2> 1648314 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[74]} 0 0
   [junit4]   2> 1648314 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[75]} 0 0
   [junit4]   2> 1648315 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[76]} 0 0
   [junit4]   2> 1648315 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[77]} 0 0
   [junit4]   2> 1648315 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[78]} 0 0
   [junit4]   2> 1648348 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[79]} 0 32
   [junit4]   2> 1648349 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[80]} 0 0
   [junit4]   2> 1648350 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[81]} 0 0
   [junit4]   2> 1648350 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[82]} 0 0
   [junit4]   2> 1648350 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[83]} 0 0
   [junit4]   2> 1648369 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[84]} 0 0
   [junit4]   2> 1648369 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[85]} 0 0
   [junit4]   2> 1648373 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[86]} 0 0
   [junit4]   2> 1648385 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[87]} 0 0
   [junit4]   2> 1648386 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1648386 INFO  (indexFetcher-2597-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 1648386 INFO  (indexFetcher-2597-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 1648386 INFO  (indexFetcher-2597-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1648386 INFO  (indexFetcher-2597-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 1648386 INFO  (indexFetcher-2597-thread-1) [    x:collection1] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
   [junit4]   2> 1648443 INFO  (indexFetcher-2597-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1648443 INFO  (indexFetcher-2597-thread-1) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@564cac2e commitCommandVersion:0
   [junit4]   2> 1648444 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[88]} 0 0
   [junit4]   2> 1648444 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[89]} 0 0
   [junit4]   2> 1648445 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[90]} 0 0
   [junit4]   2> 1648445 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[91]} 0 0
   [junit4]   2> 1648445 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[92]} 0 0
   [junit4]   2> 1648446 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[93]} 0 0
   [junit4]   2> 1648461 INFO  (indexFetcher-2597-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@15325485[collection1] main]
   [junit4]   2> 1648461 INFO  (indexFetcher-2597-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1648491 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[94]} 0 44
   [junit4]   2> 1648491 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[95]} 0 0
   [junit4]   2> 1648495 INFO  (searcherExecutor-2593-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@15325485[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1648507 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[96]} 0 0
   [junit4]   2> 1648508 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[97]} 0 0
   [junit4]   2> 1648508 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[98]} 0 0
   [junit4]   2> 1648508 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[99]} 0 0
   [junit4]   2> 1648509 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[100]} 0 0
   [junit4]   2> 1648509 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[101]} 0 0
   [junit4]   2> 1648510 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[102]} 0 0
   [junit4]   2> 1648510 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[103]} 0 0
   [junit4]   2> 1648510 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[104]} 0 0
   [junit4]   2> 1648511 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[105]} 0 0
   [junit4]   2> 1648548 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[106]} 0 0
   [junit4]   2> 1648548 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[107]} 0 0
   [junit4]   2> 1648549 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[108]} 0 0
   [junit4]   2> 1648584 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[109]} 0 32
   [junit4]   2> 1648601 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[110]} 0 0
   [junit4]   2> 1648605 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[111]} 0 0
   [junit4]   2> 1648621 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[112]} 0 0
   [junit4]   2> 1648622 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[113]} 0 0
   [junit4]   2> 1648622 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[114]} 0 0
   [junit4]   2> 1648622 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[115]} 0 0
   [junit4]   2> 1648623 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[116]} 0 0
   [junit4]   2> 1648623 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[117]} 0 0
   [junit4]   2> 1648624 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[118]} 0 0
   [junit4]   2> 1648624 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[119]} 0 0
   [junit4]   2> 1648624 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[120]} 0 0
   [junit4]   2> 1648643 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[121]} 0 17
   [junit4]   2> 1648643 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[122]} 0 0
   [junit4]   2> 1648643 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[123]} 0 0
   [junit4]   2> 1648644 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[124]} 0 0
   [junit4]   2> 1648644 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[125]} 0 0
   [junit4]   2> 1648644 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[126]} 0 0
   [junit4]   2> 1648645 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[127]} 0 0
   [junit4]   2> 1648645 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[128]} 0 0
   [junit4]   2> 1648661 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[129]} 0 0
   [junit4]   2> 1648661 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[130]} 0 0
   [junit4]   2> 1648661 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[131]} 0 0
   [junit4]   2> 1648678 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[132]} 0 0
   [junit4]   2> 1648679 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[133]} 0 0
   [junit4]   2> 1648679 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[134]} 0 0
   [junit4]   2> 1648679 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[135]} 0 0
   [junit4]   2> 1648680 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[136]} 0 0
   [junit4]   2> 1648680 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[137]} 0 0
   [junit4]   2> 1648680 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[138]} 0 0
   [junit4]   2> 1648681 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[139]} 0 0
   [junit4]   2> 1648681 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[140]} 0 0
   [junit4]   2> 1648681 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[141]} 0 0
   [junit4]   2> 1648682 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[142]} 0 0
   [junit4]   2> 1648694 INFO  (qtp1778390070-6918) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[143]} 0 0
   [junit4]   2> 1648695 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[144]} 0 0
   [junit4]   2> 1648695 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[145]} 0 0
   [junit4]   2> 1648696 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[146]} 0 0
   [junit4]   2> 1648696 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[147]} 0 0
   [junit4]   2> 1648696 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[148]} 0 0
   [junit4]   2> 1648697 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[149]} 0 0
   [junit4]   2> 1648697 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[150]} 0 0
   [junit4]   2> 1648697 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[151]} 0 0
   [junit4]   2> 1648698 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[152]} 0 0
   [junit4]   2> 1648714 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[153]} 0 16
   [junit4]   2> 1648715 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[154]} 0 0
   [junit4]   2> 1648715 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[155]} 0 0
   [junit4]   2> 1648715 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[156]} 0 0
   [junit4]   2> 1648716 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[157]} 0 0
   [junit4]   2> 1648716 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[158]} 0 0
   [junit4]   2> 1648716 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[159]} 0 0
   [junit4]   2> 1648717 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[160]} 0 0
   [junit4]   2> 1648717 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[161]} 0 0
   [junit4]   2> 1648717 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[162]} 0 0
   [junit4]   2> 1648717 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[163]} 0 0
   [junit4]   2> 1648718 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[164]} 0 0
   [junit4]   2> 1648718 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[165]} 0 0
   [junit4]   2> 1648718 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[166]} 0 0
   [junit4]   2> 1648731 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[167]} 0 0
   [junit4]   2> 1648731 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[168]} 0 0
   [junit4]   2> 1648732 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[169]} 0 0
   [junit4]   2> 1648732 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[170]} 0 0
   [junit4]   2> 1648732 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[171]} 0 0
   [junit4]   2> 1648732 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[172]} 0 0
   [junit4]   2> 1648733 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[173]} 0 0
   [junit4]   2> 1648733 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[174]} 0 0
   [junit4]   2> 1648733 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[175]} 0 0
   [junit4]   2> 1648734 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[176]} 0 0
   [junit4]   2> 1648734 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[177]} 0 0
   [junit4]   2> 1648734 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[178]} 0 0
   [junit4]   2> 1648735 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[179]} 0 0
   [junit4]   2> 1648751 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[180]} 0 0
   [junit4]   2> 1648751 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[181]} 0 0
   [junit4]   2> 1648752 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[182]} 0 0
   [junit4]   2> 1648752 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[183]} 0 0
   [junit4]   2> 1648752 INFO  (qtp1778390070-6916) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] 

[...truncated too long message...]

D69635DF]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 3420248 INFO  (searcherExecutor-7025-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@490a821c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C1)))}
   [junit4]   2> 3420248 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[42B4BD78D69635DF]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 16
   [junit4]   2> 3420249 INFO  (TEST-TestJmxIntegration.testJmxUpdate-seed#[42B4BD78D69635DF]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testJmxUpdate
   [junit4]   2> 3420360 INFO  (TEST-TestJmxIntegration.testJmxRegistration-seed#[42B4BD78D69635DF]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testJmxRegistration
   [junit4]   2> 3421399 INFO  (TEST-TestJmxIntegration.testJmxRegistration-seed#[42B4BD78D69635DF]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testJmxRegistration
   [junit4]   2> 3421399 INFO  (SUITE-TestJmxIntegration-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 3421399 INFO  (SUITE-TestJmxIntegration-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=345179059
   [junit4]   2> 3421399 INFO  (SUITE-TestJmxIntegration-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 3421399 INFO  (SUITE-TestJmxIntegration-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5fd3148a: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@31cbcd13
   [junit4]   2> 3421420 INFO  (SUITE-TestJmxIntegration-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 3421420 INFO  (SUITE-TestJmxIntegration-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@30255b8b: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@5bdce51b
   [junit4]   2> 3421436 INFO  (SUITE-TestJmxIntegration-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 3421436 INFO  (SUITE-TestJmxIntegration-seed#[42B4BD78D69635DF]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7cb3da5a: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@2f2557ea
   [junit4]   2> 3421455 INFO  (coreCloseExecutor-7030-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@69461bc9
   [junit4]   2> 3421455 INFO  (coreCloseExecutor-7030-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1766202313
   [junit4]   2> 3421455 INFO  (coreCloseExecutor-7030-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4f66f015: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@6115427e
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.core.TestJmxIntegration_42B4BD78D69635DF-001
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@6fa2f467), locale=ar-MA, timezone=America/Port-au-Prince
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=179815064,total=530055168
   [junit4]   2> NOTE: All tests run in this JVM: [BasicDistributedZkTest, ZkSolrClientTest, TestManagedSchemaAPI, TestSolrCloudWithSecureImpersonation, LargeFieldTest, SortSpecParsingTest, CoreMergeIndexesAdminHandlerTest, TestPHPSerializedResponseWriter, DebugComponentTest, MetricTriggerTest, CdcrBootstrapTest, TestHighlightDedupGrouping, MetricUtilsTest, TestLuceneIndexBackCompat, TestCollationFieldDocValues, TestHashQParserPlugin, OverseerStatusTest, DefaultValueUpdateProcessorTest, TestReplicaProperties, ExternalFileFieldSortTest, TestRuleBasedAuthorizationPlugin, SolrIndexSplitterTest, TestRebalanceLeaders, DistribDocExpirationUpdateProcessorTest, TestCloudPseudoReturnFields, TestTolerantUpdateProcessorRandomCloud, ConcurrentCreateRoutedAliasTest, UUIDFieldTest, TestSQLHandlerNonCloud, UninvertDocValuesMergePolicyTest, SmileWriterTest, TestGroupingSearch, UpdateParamsTest, CollectionTooManyReplicasTest, HdfsRecoveryZkTest, TestSmileRequest, StatsComponentTest, TestConfigSets, SynonymTokenizerTest, TestBinaryField, SolrXmlInZkTest, TestMaxTokenLenTokenizer, TestExtendedDismaxParser, TestSolrConfigHandler, TestLeaderInitiatedRecoveryThread, TestLegacyTerms, TestIndexingPerformance, RuleEngineTest, TestMergePolicyConfig, AtomicUpdateProcessorFactoryTest, DocExpirationUpdateProcessorFactoryTest, CoreSorterTest, EchoParamsTest, TestFileDictionaryLookup, TestJsonRequest, TestManagedStopFilterFactory, AutoscalingHistoryHandlerTest, TestMinMaxOnMultiValuedField, DistributedQueryElevationComponentTest, TestNestedDocsSort, XmlUpdateRequestHandlerTest, CreateCollectionCleanupTest, AssignBackwardCompatibilityTest, TestSolrFieldCacheBean, BitVectorTest, BlockJoinFacetSimpleTest, SolrIndexMetricsTest, TestSegmentSorting, TestWordDelimiterFilterFactory, SpatialHeatmapFacetsTest, HealthCheckHandlerTest, SimpleFacetsTest, LeaderFailureAfterFreshStartTest, TestDistribIDF, TriLevelCompositeIdRoutingTest, BadIndexSchemaTest, AutoAddReplicasPlanActionTest, HdfsBasicDistributedZk2Test, IndexBasedSpellCheckerTest, TestNoOpRegenerator, TestCloudInspectUtil, BinaryUpdateRequestHandlerTest, InfixSuggestersTest, FieldAnalysisRequestHandlerTest, TestDocSet, TestConfigOverlay, TestReloadDeadlock, TestClassicSimilarityFactory, TestFieldCacheSortRandom, TestSubQueryTransformerDistrib, DocumentAnalysisRequestHandlerTest, ShardRoutingTest, DistributedFacetPivotLargeTest, HttpSolrCallGetCoreTest, DeleteLastCustomShardedReplicaTest, TestLRUStatsCache, TestSolrCLIRunExample, TestFieldSortValues, UpdateLogTest, SearchHandlerTest, TestUninvertingReader, TestPolicyCloud, TestTriggerIntegration, RollingRestartTest, DistributedFacetPivotSmallTest, MoveReplicaHDFSTest, TestReloadAndDeleteDocs, TestRandomFaceting, TestChildDocTransformer, TestHashPartitioner, ReplaceNodeTest, TestSolrCoreSnapshots, JvmMetricsTest, TestStressLiveNodes, DistributedFacetSimpleRefinementLongTailTest, TestRemoteStreaming, TestCSVLoader, RequestHandlersTest, NodeMutatorTest, TestLuceneMatchVersion, TestCollationField, TestSort, TestDistribDocBasedVersion, TestConfigSetsAPIZkFailure, TestLegacyFieldCache, TestMaxScoreQueryParser, HdfsDirectoryTest, RecoveryZkTest, TestRandomDVFaceting, TestDistributedGrouping, TestRecovery, TermVectorComponentDistributedTest, ZkControllerTest, TestJoin, DistributedTermsComponentTest, SolrCoreTest, SolrCmdDistributorTest, ConvertedLegacyTest, TestFunctionQuery, BasicFunctionalityTest, DirectUpdateHandlerTest, TestIndexSearcher, SimplePostToolTest, SpellCheckCollatorTest, SuggesterFSTTest, CoreAdminHandlerTest, SolrRequestParserTest, DocValuesTest, NoCacheHeaderTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, TestAtomicUpdateErrorCases, DirectUpdateHandlerOptimizeTest, StatelessScriptUpdateProcessorFactoryTest, CacheHeaderTest, TestQueryUtils, DirectSolrSpellCheckerTest, PrimitiveFieldTypeTest, FileBasedSpellCheckerTest, TermVectorComponentTest, RequiredFieldsTest, SolrPluginUtilsTest, TestJmxIntegration]
   [junit4] Completed [442/779 (5!)] on J0 in 5.38s, 3 tests, 1 failure <<< FAILURES!

[...truncated 8589 lines...]
   [junit4] Suite: org.apache.solr.ltr.TestLTRReRankingPipeline
   [junit4]   2> 0    INFO  (SUITE-TestLTRReRankingPipeline-seed#[1C72B4201CB89142]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 7192 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[1C72B4201CB89142]) [    ] o.a.s.l.TestLTRReRankingPipeline rerank 1 documents 
   [junit4]   2> 7229 INFO  (TEST-TestLTRReRankingPipeline.testDifferentTopN-seed#[1C72B4201CB89142]) [    ] o.a.s.l.TestLTRReRankingPipeline doc 0 in pos 0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLTRReRankingPipeline -Dtests.method=testDifferentTopN -Dtests.seed=1C72B4201CB89142 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=de-GR -Dtests.timezone=Etc/GMT-3 -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.45s J0 | TestLTRReRankingPipeline.testDifferentTopN <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1.0> but was:<0.0>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1C72B4201CB89142:EDD3C67029035BD0]:0)
   [junit4]    > 	at org.apache.solr.ltr.TestLTRReRankingPipeline.testDifferentTopN(TestLTRReRankingPipeline.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@25d45a9b), locale=de-GR, timezone=Etc/GMT-3
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=234143112,total=319291392
   [junit4]   2> NOTE: All tests run in this JVM: [TestLTRReRankingPipeline]
   [junit4] Completed [1/37 (1!)] on J0 in 10.20s, 3 tests, 1 failure <<< FAILURES!

[...truncated 40406 lines...]