You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2018/09/06 03:50:08 UTC

[JENKINS] Lucene-Solr-BadApples-master-Linux (64bit/jdk-9.0.4) - Build # 88 - Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-BadApples-master-Linux/88/
Java: 64bit/jdk-9.0.4 -XX:+UseCompressedOops -XX:+UseParallelGC

2 tests failed.
FAILED:  org.apache.solr.cloud.SSLMigrationTest.test

Error Message:
Replica didn't have the proper urlScheme in the ClusterState

Stack Trace:
java.lang.AssertionError: Replica didn't have the proper urlScheme in the ClusterState
	at __randomizedtesting.SeedInfo.seed([A0E2BA5925430DF8:28B685838BBF6000]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.SSLMigrationTest.assertReplicaInformation(SSLMigrationTest.java:104)
	at org.apache.solr.cloud.SSLMigrationTest.testMigrateSSL(SSLMigrationTest.java:97)
	at org.apache.solr.cloud.SSLMigrationTest.test(SSLMigrationTest.java:61)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1008)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:983)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.testDelegationTokenRenew

Error Message:
expected:<200> but was:<403>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<403>
	at __randomizedtesting.SeedInfo.seed([A0E2BA5925430DF8:97794E471D8FD05C]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.renewDelegationToken(TestDelegationWithHadoopAuth.java:121)
	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.verifyDelegationTokenRenew(TestDelegationWithHadoopAuth.java:303)
	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.testDelegationTokenRenew(TestDelegationWithHadoopAuth.java:321)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 12938 lines...]
   [junit4] Suite: org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth
   [junit4]   2> 683031 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[A0E2BA5925430DF8]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J2/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_A0E2BA5925430DF8-001/init-core-data-001
   [junit4]   2> 683032 WARN  (SUITE-TestDelegationWithHadoopAuth-seed#[A0E2BA5925430DF8]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 683032 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[A0E2BA5925430DF8]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 683033 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[A0E2BA5925430DF8]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
   [junit4]   2> 683033 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[A0E2BA5925430DF8]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J2/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_A0E2BA5925430DF8-001/tempDir-001
   [junit4]   2> 683033 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[A0E2BA5925430DF8]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 683036 INFO  (Thread-3559) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 683036 INFO  (Thread-3559) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 683037 ERROR (Thread-3559) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 683136 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[A0E2BA5925430DF8]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:34971
   [junit4]   2> 683143 INFO  (zkConnectionManagerCallback-2013-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683147 INFO  (jetty-launcher-2010-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 9.0.4+11
   [junit4]   2> 683148 INFO  (jetty-launcher-2010-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 9.0.4+11
   [junit4]   2> 683148 INFO  (jetty-launcher-2010-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 683148 INFO  (jetty-launcher-2010-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 683148 INFO  (jetty-launcher-2010-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 683156 INFO  (jetty-launcher-2010-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 683156 INFO  (jetty-launcher-2010-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 683156 INFO  (jetty-launcher-2010-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 683156 INFO  (jetty-launcher-2010-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@697a249d{/solr,null,AVAILABLE}
   [junit4]   2> 683156 INFO  (jetty-launcher-2010-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@df1b2a7{/solr,null,AVAILABLE}
   [junit4]   2> 683157 INFO  (jetty-launcher-2010-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@61e30ccc{HTTP/1.1,[http/1.1]}{127.0.0.1:41553}
   [junit4]   2> 683157 INFO  (jetty-launcher-2010-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@780e7d46{HTTP/1.1,[http/1.1]}{127.0.0.1:42549}
   [junit4]   2> 683157 INFO  (jetty-launcher-2010-thread-1) [    ] o.e.j.s.Server Started @683198ms
   [junit4]   2> 683157 INFO  (jetty-launcher-2010-thread-2) [    ] o.e.j.s.Server Started @683198ms
   [junit4]   2> 683157 INFO  (jetty-launcher-2010-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41553}
   [junit4]   2> 683157 INFO  (jetty-launcher-2010-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42549}
   [junit4]   2> 683157 ERROR (jetty-launcher-2010-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 683157 ERROR (jetty-launcher-2010-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 683157 INFO  (jetty-launcher-2010-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 683157 INFO  (jetty-launcher-2010-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 683157 INFO  (jetty-launcher-2010-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 683157 INFO  (jetty-launcher-2010-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 683157 INFO  (jetty-launcher-2010-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-06T02:48:19.141791Z
   [junit4]   2> 683158 INFO  (jetty-launcher-2010-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 683158 INFO  (jetty-launcher-2010-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 683158 INFO  (jetty-launcher-2010-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 683158 INFO  (jetty-launcher-2010-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 683158 INFO  (jetty-launcher-2010-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-06T02:48:19.142261Z
   [junit4]   2> 683160 INFO  (zkConnectionManagerCallback-2015-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683160 INFO  (jetty-launcher-2010-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 683160 INFO  (zkConnectionManagerCallback-2017-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683163 INFO  (jetty-launcher-2010-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 683236 INFO  (jetty-launcher-2010-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34971/solr
   [junit4]   2> 683240 INFO  (zkConnectionManagerCallback-2021-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683264 INFO  (zkConnectionManagerCallback-2023-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683298 INFO  (jetty-launcher-2010-thread-2) [n:127.0.0.1:41553_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41553_solr
   [junit4]   2> 683299 INFO  (jetty-launcher-2010-thread-2) [n:127.0.0.1:41553_solr    ] o.a.s.c.Overseer Overseer (id=72175413128396804-127.0.0.1:41553_solr-n_0000000000) starting
   [junit4]   2> 683308 INFO  (zkConnectionManagerCallback-2030-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683309 INFO  (jetty-launcher-2010-thread-2) [n:127.0.0.1:41553_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34971/solr ready
   [junit4]   2> 683312 INFO  (OverseerStateUpdate-72175413128396804-127.0.0.1:41553_solr-n_0000000000) [n:127.0.0.1:41553_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:41553_solr
   [junit4]   2> 683320 INFO  (jetty-launcher-2010-thread-2) [n:127.0.0.1:41553_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41553_solr
   [junit4]   2> 683320 INFO  (jetty-launcher-2010-thread-2) [n:127.0.0.1:41553_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 683324 INFO  (jetty-launcher-2010-thread-2) [n:127.0.0.1:41553_solr    ] o.a.s.s.HadoopAuthPlugin Params: {delegation-token.token-kind=solr-dt, cookie.domain=127.0.0.1, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, type=simple, zk-dt-secret-manager.enable=true, signer.secret.provider=zookeeper}
   [junit4]   2> 683331 INFO  (zkCallback-2029-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 683342 INFO  (zkCallback-2022-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 683343 INFO  (jetty-launcher-2010-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34971/solr
   [junit4]   2> 683352 INFO  (zkConnectionManagerCallback-2034-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683357 INFO  (jetty-launcher-2010-thread-2) [n:127.0.0.1:41553_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 683359 INFO  (zkConnectionManagerCallback-2036-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683362 INFO  (jetty-launcher-2010-thread-1) [n:127.0.0.1:42549_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 683364 INFO  (jetty-launcher-2010-thread-1) [n:127.0.0.1:42549_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 683364 INFO  (jetty-launcher-2010-thread-1) [n:127.0.0.1:42549_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42549_solr
   [junit4]   2> 683367 INFO  (jetty-launcher-2010-thread-1) [n:127.0.0.1:42549_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 683368 INFO  (jetty-launcher-2010-thread-1) [n:127.0.0.1:42549_solr    ] o.a.s.s.HadoopAuthPlugin Params: {delegation-token.token-kind=solr-dt, cookie.domain=127.0.0.1, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, type=simple, zk-dt-secret-manager.enable=true, signer.secret.provider=zookeeper}
   [junit4]   2> 683368 INFO  (zkCallback-2022-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 683368 INFO  (zkCallback-2029-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 683368 INFO  (zkCallback-2035-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 683368 DEBUG (OverseerStateUpdate-72175413128396804-127.0.0.1:41553_solr-n_0000000000) [n:127.0.0.1:41553_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:42549_solr"} current state version: 0
   [junit4]   2> 683368 INFO  (jetty-launcher-2010-thread-1) [n:127.0.0.1:42549_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 683377 INFO  (jetty-launcher-2010-thread-2-EventThread) [    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 683382 INFO  (jetty-launcher-2010-thread-1-EventThread) [    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 683426 INFO  (jetty-launcher-2010-thread-1) [n:127.0.0.1:42549_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 683426 INFO  (jetty-launcher-2010-thread-2) [n:127.0.0.1:41553_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 683449 INFO  (jetty-launcher-2010-thread-2) [n:127.0.0.1:41553_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 683449 INFO  (zkConnectionManagerCallback-2043-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683450 INFO  (jetty-launcher-2010-thread-1) [n:127.0.0.1:42549_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 683450 INFO  (jetty-launcher-2010-thread-1) [n:127.0.0.1:42549_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34971/solr ready
   [junit4]   2> 683451 INFO  (jetty-launcher-2010-thread-1) [n:127.0.0.1:42549_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 683468 INFO  (jetty-launcher-2010-thread-2) [n:127.0.0.1:41553_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41553.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@12d927bf
   [junit4]   2> 683474 INFO  (jetty-launcher-2010-thread-1) [n:127.0.0.1:42549_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42549.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@12d927bf
   [junit4]   2> 683477 INFO  (jetty-launcher-2010-thread-2) [n:127.0.0.1:41553_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41553.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@12d927bf
   [junit4]   2> 683477 INFO  (jetty-launcher-2010-thread-2) [n:127.0.0.1:41553_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41553.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@12d927bf
   [junit4]   2> 683478 INFO  (jetty-launcher-2010-thread-2) [n:127.0.0.1:41553_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J2/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_A0E2BA5925430DF8-001/tempDir-001/node2/.
   [junit4]   2> 683481 INFO  (jetty-launcher-2010-thread-1) [n:127.0.0.1:42549_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42549.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@12d927bf
   [junit4]   2> 683482 INFO  (jetty-launcher-2010-thread-1) [n:127.0.0.1:42549_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42549.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@12d927bf
   [junit4]   2> 683482 INFO  (jetty-launcher-2010-thread-1) [n:127.0.0.1:42549_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J2/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_A0E2BA5925430DF8-001/tempDir-001/node1/.
   [junit4]   2> 683508 INFO  (zkConnectionManagerCallback-2047-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683509 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006b27e509000b, likely client has closed socket
   [junit4]   2> 683526 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenRenew
   [junit4]   2> 683546 WARN  (qtp1560311537-9050) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: foo is trying to renew a token with wrong password
   [junit4]   2> 683548 WARN  (qtp1560311537-9050) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 683552 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenRenew
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDelegationWithHadoopAuth -Dtests.method=testDelegationTokenRenew -Dtests.seed=A0E2BA5925430DF8 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=en-UG -Dtests.timezone=America/Port-au-Prince -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.04s J2 | TestDelegationWithHadoopAuth.testDelegationTokenRenew <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<200> but was:<403>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A0E2BA5925430DF8:97794E471D8FD05C]:0)
   [junit4]    > 	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.renewDelegationToken(TestDelegationWithHadoopAuth.java:121)
   [junit4]    > 	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.verifyDelegationTokenRenew(TestDelegationWithHadoopAuth.java:303)
   [junit4]    > 	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.testDelegationTokenRenew(TestDelegationWithHadoopAuth.java:321)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> 683569 INFO  (TEST-TestDelegationWithHadoopAuth.testZNodePaths-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testZNodePaths
   [junit4]   2> 683573 INFO  (zkConnectionManagerCallback-2050-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683574 INFO  (TEST-TestDelegationWithHadoopAuth.testZNodePaths-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testZNodePaths
   [junit4]   2> 683585 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenSolrClient-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenSolrClient
   [junit4]   2> 683598 INFO  (qtp1560311537-9053) [n:127.0.0.1:41553_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAWWsx8txigFl0NRPcQMEFAceSyYHX9vbO0FsgOtuUc3Gc-6KB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 683598 INFO  (qtp1560311537-9053) [n:127.0.0.1:41553_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAWWsx8txigFl0NRPcQMEFAceSyYHX9vbO0FsgOtuUc3Gc-6KB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 683598 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenSolrClient-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenSolrClient
   [junit4]   2> 683611 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenRenewFail
   [junit4]   2> 683614 WARN  (qtp1560311537-9140) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: bar tries to renew a token with renewer foo
   [junit4]   2> 683614 WARN  (qtp1560311537-9140) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 683616 WARN  (qtp763331096-9048) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: bar tries to renew a token with renewer foo
   [junit4]   2> 683616 WARN  (qtp763331096-9048) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 683619 WARN  (qtp1560311537-9140) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: notRenewUser tries to renew a token with renewer renewUser
   [junit4]   2> 683619 WARN  (qtp1560311537-9140) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 683620 WARN  (qtp763331096-9042) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: notRenewUser tries to renew a token with renewer renewUser
   [junit4]   2> 683620 WARN  (qtp763331096-9042) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 683621 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenRenewFail
   [junit4]   2> 683636 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenVerify
   [junit4]   2> 683643 INFO  (zkConnectionManagerCallback-2055-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683644 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 683645 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34971/solr ready
   [junit4]   2> 683646 INFO  (qtp1560311537-9049) [n:127.0.0.1:41553_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAWWsx8ukigFl0NRPpAYEFCt3JOMvAICp4clsMs7xNv3m8rZQB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 683646 INFO  (qtp1560311537-9049) [n:127.0.0.1:41553_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAWWsx8ukigFl0NRPpAYEFCt3JOMvAICp4clsMs7xNv3m8rZQB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 683659 INFO  (zkConnectionManagerCallback-2060-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683660 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 683660 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34971/solr ready
   [junit4]   2> 683662 INFO  (qtp1560311537-9057) [n:127.0.0.1:41553_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAWWsx8ukigFl0NRPpAYEFCt3JOMvAICp4clsMs7xNv3m8rZQB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 683662 INFO  (qtp1560311537-9057) [n:127.0.0.1:41553_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAWWsx8ukigFl0NRPpAYEFCt3JOMvAICp4clsMs7xNv3m8rZQB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 683666 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenVerify
   [junit4]   2> 683677 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenCancel
   [junit4]   2> 683688 ERROR (qtp1560311537-9055) [    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_7]
   [junit4]   2> 683699 INFO  (zkConnectionManagerCallback-2065-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683700 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 683701 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34971/solr ready
   [junit4]   2> 683723 ERROR (qtp763331096-9044) [    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_8]
   [junit4]   2> 683727 INFO  (zkConnectionManagerCallback-2070-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 683728 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 683728 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34971/solr ready
   [junit4]   2> 683732 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenCancel
   [junit4]   2> 683752 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancelFail-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenCancelFail
   [junit4]   2> 683759 ERROR (qtp763331096-9047) [    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_9]
   [junit4]   2> 683760 ERROR (qtp1560311537-9140) [    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_9]
   [junit4]   2> 683765 ERROR (qtp763331096-9047) [    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_10]
   [junit4]   2> 683766 ERROR (qtp1560311537-9140) [    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_10]
   [junit4]   2> 683766 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancelFail-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenCancelFail
   [junit4]   2> 683776 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationOtherOp-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationOtherOp
   [junit4]   2> 683778 INFO  (qtp1560311537-9056) [n:127.0.0.1:41553_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params op=someSolrOperation&action=LIST&user.name=foo&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 683778 INFO  (qtp1560311537-9056) [n:127.0.0.1:41553_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={op=someSolrOperation&action=LIST&user.name=foo&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 683779 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationOtherOp-seed#[A0E2BA5925430DF8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationOtherOp
   [junit4]   2> 683781 INFO  (jetty-closer-2011-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@780e7d46{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 683781 INFO  (jetty-closer-2011-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@61e30ccc{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 683781 INFO  (jetty-closer-2011-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=60679442
   [junit4]   2> 683781 INFO  (jetty-closer-2011-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 683781 INFO  (jetty-closer-2011-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1a100cf4: rootName = solr_41553, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@5a8abb99
   [junit4]   2> 683781 INFO  (jetty-closer-2011-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1315266
   [junit4]   2> 683782 INFO  (jetty-closer-2011-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 683782 INFO  (jetty-closer-2011-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@53dc7a1b: rootName = solr_42549, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@3f5f1a67
   [junit4]   2> 683788 INFO  (jetty-closer-2011-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 683788 INFO  (jetty-closer-2011-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1624d44b: rootName = solr_42549, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@ecac953
   [junit4]   2> 683788 INFO  (jetty-closer-2011-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 683788 INFO  (jetty-closer-2011-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@63ebfed2: rootName = solr_41553, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@ecac953
   [junit4]   2> 683791 INFO  (jetty-closer-2011-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 683791 INFO  (jetty-closer-2011-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6fbc8c9c: rootName = solr_42549, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@650c7ae0
   [junit4]   2> 683792 INFO  (jetty-closer-2011-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 683792 INFO  (jetty-closer-2011-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7b0cd4b: rootName = solr_41553, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@650c7ae0
   [junit4]   2> 683792 INFO  (jetty-closer-2011-thread-1) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:41553_solr
   [junit4]   2> 683792 INFO  (jetty-closer-2011-thread-2) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:42549_solr
   [junit4]   2> 683793 INFO  (jetty-closer-2011-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 683793 INFO  (zkCallback-2029-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 683793 INFO  (jetty-closer-2011-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 683793 INFO  (zkCallback-2022-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 683793 INFO  (jetty-closer-2011-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 683793 INFO  (zkCallback-2042-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 683793 INFO  (jetty-closer-2011-thread-1) [    ] o.a.s.c.Overseer Overseer (id=72175413128396804-127.0.0.1:41553_solr-n_0000000000) closing
   [junit4]   2> 683793 WARN  (OverseerAutoScalingTriggerThread-72175413128396804-127.0.0.1:41553_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 683794 INFO  (OverseerStateUpdate-72175413128396804-127.0.0.1:41553_solr-n_0000000000) [n:127.0.0.1:41553_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:41553_solr
   [junit4]   2> 683794 INFO  (zkCallback-2035-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 683795 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006b27e5090005, likely client has closed socket
   [junit4]   2> 683795 ERROR (Thread[Thread-3576,5,TGRP-TestDelegationWithHadoopAuth]) [    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 683795 ERROR (Thread[Thread-3578,5,TGRP-TestDelegationWithHadoopAuth]) [    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 683798 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006b27e5090008, likely client has closed socket
   [junit4]   2> 683798 INFO  (jetty-closer-2011-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@697a249d{/solr,null,UNAVAILABLE}
   [junit4]   2> 683798 INFO  (jetty-closer-2011-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@df1b2a7{/solr,null,UNAVAILABLE}
   [junit4]   2> 683798 INFO  (jetty-closer-2011-thread-2) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 683798 INFO  (jetty-closer-2011-thread-1) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 683799 ERROR (SUITE-TestDelegationWithHadoopAuth-seed#[A0E2BA5925430DF8]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 683799 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[A0E2BA5925430DF8]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34971 34971
   [junit4]   2> 683826 INFO  (Thread-3559) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34971 34971
   [junit4]   2> 683826 WARN  (Thread-3559) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	8	/solr/aliases.json
   [junit4]   2> 	8	/solr/clusterprops.json
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	8	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_7
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_6
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_9
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_8
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_5
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_10
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	8	/solr/live_nodes
   [junit4]   2> 	8	/solr/collections
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J2/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_A0E2BA5925430DF8-001
   [junit4]   2> Sep 06, 2018 2:48:19 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 8 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=1170, maxMBSortInHeap=6.034683604536931, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@1ec81329), locale=en-UG, timezone=America/Port-au-Prince
   [junit4]   2> NOTE: Linux 4.15.0-32-generic amd64/Oracle Corporation 9.0.4 (64-bit)/cpus=8,threads=1,free=296199072,total=494927872
   [junit4]   2> NOTE: All tests run in this JVM: [DocumentBuilderTest, CdcrUpdateLogTest, ZkSolrClientTest, MetricsHistoryHandlerTest, CleanupOldIndexTest, TestZkAclsWithHadoopAuth, TestNRTOpen, BlockJoinFacetSimpleTest, NodeAddedTriggerIntegrationTest, V2StandaloneTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestConfigSetsAPIExclusivity, TransactionLogTest, TestIntervalFaceting, EmbeddedSolrNoSerializeTest, TestIndexSearcher, EnumFieldTest, TestXmlQParser, LeaderTragicEventTest, TestEmbeddedSolrServerAdminHandler, TestHdfsUpdateLog, TestCollectionsAPIViaSolrCloudCluster, TestAnalyzeInfixSuggestions, ScheduledMaintenanceTriggerTest, MergeStrategyTest, BasicZkTest, CheckHdfsIndexTest, TestFoldingMultitermQuery, TestSolrJ, TestStressReorder, TestRawTransformer, TestRecovery, FacetPivotSmallTest, CdcrRequestHandlerTest, DistributedFacetPivotSmallTest, TestBackupRepositoryFactory, SchemaVersionSpecificBehaviorTest, CollectionTooManyReplicasTest, TestFieldSortValues, DeleteInactiveReplicaTest, RestartWhileUpdatingTest, TestNumericTokenStream, TestCollapseQParserPlugin, TestClusterStateMutator, TestSolr4Spatial2, SaslZkACLProviderTest, BasicAuthStandaloneTest, RequestLoggingTest, ChaosMonkeySafeLeaderTest, TestFileDictionaryLookup, TestAuthenticationFramework, TestPrepRecovery, AddReplicaTest, TestSimpleQParserPlugin, ConvertedLegacyTest, SOLR749Test, TestConfig, OpenCloseCoreStressTest, TestOverriddenPrefixQueryForCustomFieldType, AsyncCallRequestStatusResponseTest, TestDFRSimilarityFactory, TestPKIAuthenticationPlugin, TestJoin, ReplaceNodeNoTargetTest, TestBinaryResponseWriter, TemplateUpdateProcessorTest, CdcrBootstrapTest, SolrRequestParserTest, TestHighFrequencyDictionaryFactory, TestDirectoryFactory, TestDeleteCollectionOnDownNodes, TestObjectReleaseTracker, TestCharFilters, TestDelegationWithHadoopAuth]
   [junit4] Completed [165/832 (1!)] on J2 in 1.56s, 8 tests, 1 failure <<< FAILURES!

[...truncated 95 lines...]
   [junit4] Suite: org.apache.solr.cloud.SSLMigrationTest
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001/init-core-data-001
   [junit4]   2> 790161 WARN  (SUITE-SSLMigrationTest-seed#[A0E2BA5925430DF8]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=160 numCloses=160
   [junit4]   2> 790161 INFO  (SUITE-SSLMigrationTest-seed#[A0E2BA5925430DF8]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 790162 INFO  (SUITE-SSLMigrationTest-seed#[A0E2BA5925430DF8]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl="None")
   [junit4]   2> 790162 INFO  (SUITE-SSLMigrationTest-seed#[A0E2BA5925430DF8]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 790162 INFO  (SUITE-SSLMigrationTest-seed#[A0E2BA5925430DF8]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 790167 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 790176 INFO  (Thread-1130) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 790176 INFO  (Thread-1130) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 790178 ERROR (Thread-1130) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 790276 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.ZkTestServer start zk server on port:45749
   [junit4]   2> 790278 INFO  (zkConnectionManagerCallback-1301-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 790301 INFO  (zkConnectionManagerCallback-1303-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 790307 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 790309 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 790310 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 790311 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 790312 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 790313 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 790314 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 790315 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 790317 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 790317 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 790318 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 790319 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 790439 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 9.0.4+11
   [junit4]   2> 790449 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 790449 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 790449 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 790450 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5a75bf71{/,null,AVAILABLE}
   [junit4]   2> 790451 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3140d6cc{HTTP/1.1,[http/1.1]}{127.0.0.1:38297}
   [junit4]   2> 790451 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.Server Started @790490ms
   [junit4]   2> 790451 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, solr.data.dir=/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001/tempDir-001/control/data, hostPort=38297, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001/control-001/cores}
   [junit4]   2> 790451 ERROR (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 790451 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 790451 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 790451 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 790451 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 790452 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-06T02:50:06.438994Z
   [junit4]   2> 790461 INFO  (zkConnectionManagerCallback-1305-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 790462 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 790462 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001/control-001/solr.xml
   [junit4]   2> 790466 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 790466 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 790466 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5601a956, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 790514 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45749/solr
   [junit4]   2> 790515 INFO  (zkConnectionManagerCallback-1309-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 790518 INFO  (zkConnectionManagerCallback-1311-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 790614 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [n:127.0.0.1:38297_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38297_
   [junit4]   2> 790615 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [n:127.0.0.1:38297_    ] o.a.s.c.Overseer Overseer (id=72175420150185988-127.0.0.1:38297_-n_0000000000) starting
   [junit4]   2> 790622 INFO  (zkConnectionManagerCallback-1318-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 790625 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [n:127.0.0.1:38297_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45749/solr ready
   [junit4]   2> 790626 INFO  (OverseerStateUpdate-72175420150185988-127.0.0.1:38297_-n_0000000000) [n:127.0.0.1:38297_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:38297_
   [junit4]   2> 790626 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [n:127.0.0.1:38297_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38297_
   [junit4]   2> 790632 INFO  (zkCallback-1310-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 790645 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [n:127.0.0.1:38297_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 790645 INFO  (zkCallback-1317-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 790659 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [n:127.0.0.1:38297_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5601a956
   [junit4]   2> 790670 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [n:127.0.0.1:38297_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5601a956
   [junit4]   2> 790670 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [n:127.0.0.1:38297_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5601a956
   [junit4]   2> 790672 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [n:127.0.0.1:38297_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001/control-001/cores
   [junit4]   2> 790722 INFO  (zkConnectionManagerCallback-1324-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 790723 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 790723 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45749/solr ready
   [junit4]   2> 790725 INFO  (qtp1367683569-5213) [n:127.0.0.1:38297_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:38297_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 790727 INFO  (OverseerThreadFactory-2174-thread-1-processing-n:127.0.0.1:38297_) [n:127.0.0.1:38297_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 790832 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 790833 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 792309 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 792320 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 792390 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 792406 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 792407 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5601a956
   [junit4]   2> 792407 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 792407 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 792409 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=36, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=9.65625, floorSegmentMB=0.83203125, forceMergeDeletesPctAllowed=16.79811163646596, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=48.76959414056233
   [junit4]   2> 792410 WARN  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 792452 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 792452 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 792453 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 792453 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 792454 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 792454 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5f04a233[control_collection_shard1_replica_n1] main]
   [junit4]   2> 792455 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 792455 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 792456 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 792457 INFO  (searcherExecutor-2179-thread-1-processing-n:127.0.0.1:38297_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@5f04a233[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 792457 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1610824766921375744
   [junit4]   2> 792461 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 792462 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 792462 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 792462 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:38297/control_collection_shard1_replica_n1/
   [junit4]   2> 792462 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 792462 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:38297/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 792462 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 792464 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:38297/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 792566 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 792569 INFO  (qtp1367683569-5209) [n:127.0.0.1:38297_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1736
   [junit4]   2> 792571 INFO  (qtp1367683569-5213) [n:127.0.0.1:38297_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 792667 INFO  (zkCallback-1310-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 792729 INFO  (OverseerCollectionConfigSetProcessor-72175420150185988-127.0.0.1:38297_-n_0000000000) [n:127.0.0.1:38297_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 793571 INFO  (qtp1367683569-5213) [n:127.0.0.1:38297_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:38297_&wt=javabin&version=2} status=0 QTime=2845
   [junit4]   2> 793574 INFO  (zkConnectionManagerCallback-1329-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 793575 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 793576 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45749/solr ready
   [junit4]   2> 793576 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 793576 INFO  (qtp1367683569-5208) [n:127.0.0.1:38297_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 793578 INFO  (OverseerThreadFactory-2174-thread-2-processing-n:127.0.0.1:38297_) [n:127.0.0.1:38297_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 793779 WARN  (OverseerThreadFactory-2174-thread-2-processing-n:127.0.0.1:38297_) [n:127.0.0.1:38297_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 793781 INFO  (qtp1367683569-5208) [n:127.0.0.1:38297_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 793781 INFO  (qtp1367683569-5208) [n:127.0.0.1:38297_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=204
   [junit4]   2> 794558 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001/shard-1-001 of type NRT
   [junit4]   2> 794558 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 9.0.4+11
   [junit4]   2> 794559 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 794559 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 794559 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 794559 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2f4c885f{/,null,AVAILABLE}
   [junit4]   2> 794559 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1b9dda76{HTTP/1.1,[http/1.1]}{127.0.0.1:36289}
   [junit4]   2> 794559 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.Server Started @794599ms
   [junit4]   2> 794559 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, solrconfig=solrconfig.xml, solr.data.dir=/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001/tempDir-001/jetty1, hostPort=36289, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001/shard-1-001/cores}
   [junit4]   2> 794560 ERROR (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 794560 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 794560 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 794560 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 794560 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 794560 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-06T02:50:10.547347Z
   [junit4]   2> 794561 INFO  (zkConnectionManagerCallback-1331-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 794562 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 794562 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001/shard-1-001/solr.xml
   [junit4]   2> 794564 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 794564 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 794565 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5601a956, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 794649 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45749/solr
   [junit4]   2> 794651 INFO  (zkConnectionManagerCallback-1335-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 794652 INFO  (zkConnectionManagerCallback-1337-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 794656 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 794658 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 794658 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36289_
   [junit4]   2> 794659 INFO  (zkCallback-1317-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 794659 INFO  (zkCallback-1328-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 794659 INFO  (zkCallback-1336-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 794661 INFO  (zkCallback-1310-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 794671 INFO  (zkConnectionManagerCallback-1344-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 794672 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 794673 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45749/solr ready
   [junit4]   2> 794673 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 794689 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5601a956
   [junit4]   2> 794697 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5601a956
   [junit4]   2> 794697 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5601a956
   [junit4]   2> 794699 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001/shard-1-001/cores
   [junit4]   2> 794721 INFO  (qtp2086086519-5270) [n:127.0.0.1:36289_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:36289_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 794722 INFO  (OverseerCollectionConfigSetProcessor-72175420150185988-127.0.0.1:38297_-n_0000000000) [n:127.0.0.1:38297_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 794722 INFO  (OverseerThreadFactory-2174-thread-3-processing-n:127.0.0.1:38297_) [n:127.0.0.1:38297_ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:36289_ for creating new replica of shard shard2
   [junit4]   2> 794724 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_    x:collection1_shard2_replica_n41] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 795790 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 795809 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema [collection1_shard2_replica_n41] Schema name=test
   [junit4]   2> 795876 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 795887 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using configuration from collection collection1, trusted=true
   [junit4]   2> 795887 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n41' (registry 'solr.core.collection1.shard2.replica_n41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5601a956
   [junit4]   2> 795888 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 795888 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [[collection1_shard2_replica_n41] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001/shard-1-001/cores/collection1_shard2_replica_n41], dataDir=[/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
   [junit4]   2> 795890 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=36, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=9.65625, floorSegmentMB=0.83203125, forceMergeDeletesPctAllowed=16.79811163646596, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=48.76959414056233
   [junit4]   2> 795892 WARN  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 795938 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 795938 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 795939 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 795939 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 795940 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 795941 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.SolrIndexSearcher Opening [Searcher@3f168a4a[collection1_shard2_replica_n41] main]
   [junit4]   2> 795941 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 795942 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 795942 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 795942 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1610824770575663104
   [junit4]   2> 795943 INFO  (searcherExecutor-2193-thread-1-processing-n:127.0.0.1:36289_ x:collection1_shard2_replica_n41 c:collection1 s:shard2) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [collection1_shard2_replica_n41] Registered new searcher Searcher@3f168a4a[collection1_shard2_replica_n41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 795946 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node42=0}, version=0}
   [junit4]   2> 795947 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 795947 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 795947 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:36289/collection1_shard2_replica_n41/
   [junit4]   2> 795947 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 795947 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy http://127.0.0.1:36289/collection1_shard2_replica_n41/ has no replicas
   [junit4]   2> 795947 INFO  (qtp2086086519-5265) [n:127.0.0.1:36289_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election,

[...truncated too long message...]

apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1309) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1265) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1149) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1080) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:187) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) ~[?:?]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) ~[?:?]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:844) [?:?]
   [junit4]   2> 809821 ERROR (coreZkRegister-2290-thread-1-processing-n:127.0.0.1:33037_ x:collection1_shard1_replica_n47 c:collection1 s:shard1 r:core_node48) [n:127.0.0.1:33037_ c:collection1 s:shard1 r:core_node48 x:collection1_shard1_replica_n47] o.a.s.c.ZkContainer :org.apache.solr.common.SolrException: Error getting leader from zk for shard shard1
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1298)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1149)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1080)
   [junit4]   2> 	at org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:187)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
   [junit4]   2> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
   [junit4]   2> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: CoreContainer is closed
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1342)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1309)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1265)
   [junit4]   2> 	... 7 more
   [junit4]   2> 
   [junit4]   2> 809822 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3d7e1165{SSL,[ssl, http/1.1]}{127.0.0.1:33037}
   [junit4]   2> 809822 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@10042b94{/,null,UNAVAILABLE}
   [junit4]   2> 809822 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 809822 ERROR (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 809823 INFO  (TEST-SSLMigrationTest.test-seed#[A0E2BA5925430DF8]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45749 45749
   [junit4]   2> 809853 INFO  (Thread-1130) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45749 45749
   [junit4]   2> 809853 WARN  (Thread-1130) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	20	/solr/aliases.json
   [junit4]   2> 	12	/solr/collections/collection1/state.json
   [junit4]   2> 	9	/solr/security.json
   [junit4]   2> 	9	/solr/configs/conf1
   [junit4]   2> 	9	/solr/collections/collection1/terms/shard1
   [junit4]   2> 	9	/solr/collections/collection1/terms/shard2
   [junit4]   2> 	2	/solr/collections/control_collection/terms/shard1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	20	/solr/clusterprops.json
   [junit4]   2> 	20	/solr/clusterstate.json
   [junit4]   2> 	5	/solr/autoscaling.json
   [junit4]   2> 	5	/solr/overseer_elect/election/72175420150185988-127.0.0.1:38297_-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	20	/solr/live_nodes
   [junit4]   2> 	20	/solr/collections
   [junit4]   2> 	5	/solr/overseer/queue
   [junit4]   2> 	5	/solr/autoscaling/events/.scheduled_maintenance
   [junit4]   2> 	5	/solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 	5	/solr/overseer/collection-queue-work
   [junit4]   2> 	5	/solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SSLMigrationTest -Dtests.method=test -Dtests.seed=A0E2BA5925430DF8 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=bm-ML -Dtests.timezone=Asia/Katmandu -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 19.7s J1 | SSLMigrationTest.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Replica didn't have the proper urlScheme in the ClusterState
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A0E2BA5925430DF8:28B685838BBF6000]:0)
   [junit4]    > 	at org.apache.solr.cloud.SSLMigrationTest.assertReplicaInformation(SSLMigrationTest.java:104)
   [junit4]    > 	at org.apache.solr.cloud.SSLMigrationTest.testMigrateSSL(SSLMigrationTest.java:97)
   [junit4]    > 	at org.apache.solr.cloud.SSLMigrationTest.test(SSLMigrationTest.java:61)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1008)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:983)
   [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.SSLMigrationTest_A0E2BA5925430DF8-001
   [junit4]   2> Sep 06, 2018 2:50:25 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@1290ceb6), locale=bm-ML, timezone=Asia/Katmandu
   [junit4]   2> NOTE: Linux 4.15.0-32-generic amd64/Oracle Corporation 9.0.4 (64-bit)/cpus=8,threads=1,free=206571848,total=481296384
   [junit4]   2> NOTE: All tests run in this JVM: [DirectUpdateHandlerTest, AnalysisErrorHandlingTest, ZkShardTermsTest, MetricsHandlerTest, TestSolrCloudWithHadoopAuthPlugin, TestMergePolicyConfig, BlockJoinFacetRandomTest, MetricTriggerTest, V2ApiIntegrationTest, FieldMutatingUpdateProcessorTest, TestConfigSetsAPI, TestUpdate, TestFaceting, JavabinLoaderTest, TestHashQParserPlugin, DocValuesTest, TestValueSourceCache, LeaderFailureAfterFreshStartTest, TestLuceneIndexBackCompat, TestExceedMaxTermLength, TestCollectionAPI, SuggesterWFSTTest, RestoreTriggerStateTest, LargeFieldTest, BasicDistributedZkTest, WrapperMergePolicyFactoryTest, TestFiltering, TestSolrFieldCacheBean, TestStressRecovery, TestRawResponseWriter, TestRealTimeGet, DistributedTermsComponentTest, CdcrReplicationHandlerTest, DistributedFacetPivotSmallAdvancedTest, SolrCoreTest, SchemaApiFailureTest, CollectionReloadTest, TestFastLRUCache, CreateRoutedAliasTest, ReplicationFactorTest, TestNumericRangeQuery64, TestAddFieldRealTimeGet, StressHdfsTest, TestSolr4Spatial, SSLMigrationTest]
   [junit4] Completed [193/832 (2!)] on J1 in 20.47s, 1 test, 1 failure <<< FAILURES!

[...truncated 44244 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-BadApples-master-Linux/88/consoleText

[repro] Revision: 37375ae6008a069ce1362e287dfd55bed0a27f7f

[repro] Repro line:  ant test  -Dtestcase=TestDelegationWithHadoopAuth -Dtests.method=testDelegationTokenRenew -Dtests.seed=A0E2BA5925430DF8 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=en-UG -Dtests.timezone=America/Port-au-Prince -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] Repro line:  ant test  -Dtestcase=SSLMigrationTest -Dtests.method=test -Dtests.seed=A0E2BA5925430DF8 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=bm-ML -Dtests.timezone=Asia/Katmandu -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       TestDelegationWithHadoopAuth
[repro]       SSLMigrationTest
[repro] ant compile-test

[...truncated 3428 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=10 -Dtests.class="*.TestDelegationWithHadoopAuth|*.SSLMigrationTest" -Dtests.showOutput=onerror  -Dtests.seed=A0E2BA5925430DF8 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=en-UG -Dtests.timezone=America/Port-au-Prince -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[...truncated 132 lines...]
[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.SSLMigrationTest
[repro]   0/5 failed: org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth
[repro] Exiting with code 0

[...truncated 40 lines...]

[JENKINS] Lucene-Solr-BadApples-master-Linux (64bit/jdk1.8.0_172) - Build # 90 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-BadApples-master-Linux/90/
Java: 64bit/jdk1.8.0_172 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testBasic

Error Message:
last state: DocCollection(testBasic//clusterstate.json/959)={   "replicationFactor":"1",   "pullReplicas":"5",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"5",   "tlogReplicas":"5",   "autoCreated":"true",   "shards":{     "shard2":{       "replicas":{         "core_node29":{           "core":"testBasic_shard2_replica_p29",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10150_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node18":{           "core":"testBasic_shard2_replica_n18",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10054_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node28":{           "core":"testBasic_shard2_replica_p28",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10155_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node17":{           "core":"testBasic_shard2_replica_n17",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10064_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node19":{           "core":"testBasic_shard2_replica_n19",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10085_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node21":{           "core":"testBasic_shard2_replica_t21",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10140_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node20":{           "core":"testBasic_shard2_replica_n20",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10108_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node23":{           "core":"testBasic_shard2_replica_t23",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10052_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node22":{           "core":"testBasic_shard2_replica_t22",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10093_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node25":{           "core":"testBasic_shard2_replica_t25",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10136_solr",           "state":"down",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node24":{           "core":"testBasic_shard2_replica_t24",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10042_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node16":{           "core":"testBasic_shard2_replica_n16",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10114_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node27":{           "core":"testBasic_shard2_replica_p27",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10156_solr",           "state":"down",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node26":{           "core":"testBasic_shard2_replica_p26",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10087_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node30":{           "core":"testBasic_shard2_replica_p30",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10133_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"b3330000-e665ffff",       "state":"active"},     "shard3":{       "replicas":{         "core_node39":{           "core":"testBasic_shard3_replica_t39",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10152_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node32":{           "core":"testBasic_shard3_replica_n32",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10149_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node43":{           "core":"testBasic_shard3_replica_p43",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10137_solr",           "state":"down",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node31":{           "core":"testBasic_shard3_replica_n31",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10061_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node42":{           "core":"testBasic_shard3_replica_p42",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10126_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node34":{           "core":"testBasic_shard3_replica_n34",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10164_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node45":{           "core":"testBasic_shard3_replica_p45",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10027_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node44":{           "core":"testBasic_shard3_replica_p44",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10113_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node33":{           "core":"testBasic_shard3_replica_n33",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10145_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node36":{           "core":"testBasic_shard3_replica_t36",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10107_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node35":{           "core":"testBasic_shard3_replica_n35",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10045_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node38":{           "core":"testBasic_shard3_replica_t38",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10041_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node37":{           "core":"testBasic_shard3_replica_t37",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10160_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node41":{           "core":"testBasic_shard3_replica_p41",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10079_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node40":{           "core":"testBasic_shard3_replica_t40",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10131_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"e6660000-1998ffff",       "state":"active"},     "shard4":{       "replicas":{         "core_node54":{           "core":"testBasic_shard4_replica_t54",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10071_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node53":{           "core":"testBasic_shard4_replica_t53",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10146_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node56":{           "core":"testBasic_shard4_replica_p56",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10090_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node55":{           "core":"testBasic_shard4_replica_t55",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10161_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node47":{           "core":"testBasic_shard4_replica_n47",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10088_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node58":{           "core":"testBasic_shard4_replica_p58",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10092_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node57":{           "core":"testBasic_shard4_replica_p57",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10044_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node46":{           "core":"testBasic_shard4_replica_n46",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10040_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node49":{           "core":"testBasic_shard4_replica_n49",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10038_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node48":{           "core":"testBasic_shard4_replica_n48",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10076_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node59":{           "core":"testBasic_shard4_replica_p59",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10083_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node50":{           "core":"testBasic_shard4_replica_n50",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10067_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node60":{           "core":"testBasic_shard4_replica_p60",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10101_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node52":{           "core":"testBasic_shard4_replica_t52",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10030_solr",           "state":"down",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node51":{           "core":"testBasic_shard4_replica_t51",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10143_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"19990000-4ccbffff",       "state":"active"},     "shard5":{       "replicas":{         "core_node65":{           "core":"testBasic_shard5_replica_n65",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10123_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node75":{           "core":"testBasic_shard5_replica_p75",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10130_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node64":{           "core":"testBasic_shard5_replica_n64",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10154_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node67":{           "core":"testBasic_shard5_replica_t67",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10048_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node66":{           "core":"testBasic_shard5_replica_t66",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10029_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node69":{           "core":"testBasic_shard5_replica_t69",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10127_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node68":{           "core":"testBasic_shard5_replica_t68",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10058_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node70":{           "core":"testBasic_shard5_replica_t70",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10111_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node72":{           "core":"testBasic_shard5_replica_p72",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10124_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node61":{           "core":"testBasic_shard5_replica_n61",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10043_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node71":{           "core":"testBasic_shard5_replica_p71",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10086_solr",           "state":"down",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node74":{           "core":"testBasic_shard5_replica_p74",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10134_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node63":{           "core":"testBasic_shard5_replica_n63",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10099_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node62":{           "core":"testBasic_shard5_replica_n62",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10135_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node73":{           "core":"testBasic_shard5_replica_p73",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10098_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"4ccc0000-7fffffff",       "state":"active"},     "shard1":{       "replicas":{         "core_node1":{           "core":"testBasic_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10142_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node10":{           "core":"testBasic_shard1_replica_t10",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10089_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node12":{           "core":"testBasic_shard1_replica_p12",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10039_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node11":{           "core":"testBasic_shard1_replica_p11",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10025_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node14":{           "core":"testBasic_shard1_replica_p14",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10148_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node13":{           "core":"testBasic_shard1_replica_p13",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10074_solr",           "state":"down",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node15":{           "core":"testBasic_shard1_replica_p15",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10159_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node6":{           "core":"testBasic_shard1_replica_t6",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10139_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node7":{           "core":"testBasic_shard1_replica_t7",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10068_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node8":{           "core":"testBasic_shard1_replica_t8",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10028_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node9":{           "core":"testBasic_shard1_replica_t9",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10080_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node2":{           "core":"testBasic_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10151_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node3":{           "core":"testBasic_shard1_replica_n3",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10055_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node4":{           "core":"testBasic_shard1_replica_n4",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10129_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node5":{           "core":"testBasic_shard1_replica_n5",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10153_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"80000000-b332ffff",       "state":"active"}}}

Stack Trace:
java.util.concurrent.TimeoutException: last state: DocCollection(testBasic//clusterstate.json/959)={
  "replicationFactor":"1",
  "pullReplicas":"5",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"5",
  "tlogReplicas":"5",
  "autoCreated":"true",
  "shards":{
    "shard2":{
      "replicas":{
        "core_node29":{
          "core":"testBasic_shard2_replica_p29",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10150_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node18":{
          "core":"testBasic_shard2_replica_n18",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10054_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node28":{
          "core":"testBasic_shard2_replica_p28",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10155_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node17":{
          "core":"testBasic_shard2_replica_n17",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10064_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node19":{
          "core":"testBasic_shard2_replica_n19",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10085_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node21":{
          "core":"testBasic_shard2_replica_t21",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10140_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node20":{
          "core":"testBasic_shard2_replica_n20",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10108_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node23":{
          "core":"testBasic_shard2_replica_t23",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10052_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node22":{
          "core":"testBasic_shard2_replica_t22",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10093_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node25":{
          "core":"testBasic_shard2_replica_t25",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10136_solr",
          "state":"down",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node24":{
          "core":"testBasic_shard2_replica_t24",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10042_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node16":{
          "core":"testBasic_shard2_replica_n16",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10114_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node27":{
          "core":"testBasic_shard2_replica_p27",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10156_solr",
          "state":"down",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node26":{
          "core":"testBasic_shard2_replica_p26",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10087_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node30":{
          "core":"testBasic_shard2_replica_p30",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10133_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"b3330000-e665ffff",
      "state":"active"},
    "shard3":{
      "replicas":{
        "core_node39":{
          "core":"testBasic_shard3_replica_t39",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10152_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node32":{
          "core":"testBasic_shard3_replica_n32",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10149_solr",
          "state":"down",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node43":{
          "core":"testBasic_shard3_replica_p43",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10137_solr",
          "state":"down",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node31":{
          "core":"testBasic_shard3_replica_n31",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10061_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node42":{
          "core":"testBasic_shard3_replica_p42",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10126_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node34":{
          "core":"testBasic_shard3_replica_n34",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10164_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node45":{
          "core":"testBasic_shard3_replica_p45",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10027_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node44":{
          "core":"testBasic_shard3_replica_p44",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10113_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node33":{
          "core":"testBasic_shard3_replica_n33",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10145_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node36":{
          "core":"testBasic_shard3_replica_t36",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10107_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node35":{
          "core":"testBasic_shard3_replica_n35",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10045_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node38":{
          "core":"testBasic_shard3_replica_t38",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10041_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node37":{
          "core":"testBasic_shard3_replica_t37",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10160_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node41":{
          "core":"testBasic_shard3_replica_p41",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10079_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node40":{
          "core":"testBasic_shard3_replica_t40",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10131_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"e6660000-1998ffff",
      "state":"active"},
    "shard4":{
      "replicas":{
        "core_node54":{
          "core":"testBasic_shard4_replica_t54",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10071_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node53":{
          "core":"testBasic_shard4_replica_t53",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10146_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node56":{
          "core":"testBasic_shard4_replica_p56",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10090_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node55":{
          "core":"testBasic_shard4_replica_t55",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10161_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node47":{
          "core":"testBasic_shard4_replica_n47",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10088_solr",
          "state":"down",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node58":{
          "core":"testBasic_shard4_replica_p58",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10092_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node57":{
          "core":"testBasic_shard4_replica_p57",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10044_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node46":{
          "core":"testBasic_shard4_replica_n46",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10040_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node49":{
          "core":"testBasic_shard4_replica_n49",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10038_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node48":{
          "core":"testBasic_shard4_replica_n48",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10076_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node59":{
          "core":"testBasic_shard4_replica_p59",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10083_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node50":{
          "core":"testBasic_shard4_replica_n50",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10067_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node60":{
          "core":"testBasic_shard4_replica_p60",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10101_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node52":{
          "core":"testBasic_shard4_replica_t52",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10030_solr",
          "state":"down",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node51":{
          "core":"testBasic_shard4_replica_t51",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10143_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"19990000-4ccbffff",
      "state":"active"},
    "shard5":{
      "replicas":{
        "core_node65":{
          "core":"testBasic_shard5_replica_n65",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10123_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node75":{
          "core":"testBasic_shard5_replica_p75",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10130_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node64":{
          "core":"testBasic_shard5_replica_n64",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10154_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node67":{
          "core":"testBasic_shard5_replica_t67",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10048_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node66":{
          "core":"testBasic_shard5_replica_t66",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10029_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node69":{
          "core":"testBasic_shard5_replica_t69",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10127_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node68":{
          "core":"testBasic_shard5_replica_t68",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10058_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node70":{
          "core":"testBasic_shard5_replica_t70",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10111_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node72":{
          "core":"testBasic_shard5_replica_p72",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10124_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node61":{
          "core":"testBasic_shard5_replica_n61",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10043_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node71":{
          "core":"testBasic_shard5_replica_p71",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10086_solr",
          "state":"down",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node74":{
          "core":"testBasic_shard5_replica_p74",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10134_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node63":{
          "core":"testBasic_shard5_replica_n63",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10099_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node62":{
          "core":"testBasic_shard5_replica_n62",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10135_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node73":{
          "core":"testBasic_shard5_replica_p73",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10098_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"4ccc0000-7fffffff",
      "state":"active"},
    "shard1":{
      "replicas":{
        "core_node1":{
          "core":"testBasic_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10142_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node10":{
          "core":"testBasic_shard1_replica_t10",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10089_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node12":{
          "core":"testBasic_shard1_replica_p12",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10039_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node11":{
          "core":"testBasic_shard1_replica_p11",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10025_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node14":{
          "core":"testBasic_shard1_replica_p14",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10148_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node13":{
          "core":"testBasic_shard1_replica_p13",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10074_solr",
          "state":"down",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node15":{
          "core":"testBasic_shard1_replica_p15",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10159_solr",
          "state":"active",
          "type":"PULL",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node6":{
          "core":"testBasic_shard1_replica_t6",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10139_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node7":{
          "core":"testBasic_shard1_replica_t7",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10068_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node8":{
          "core":"testBasic_shard1_replica_t8",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10028_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node9":{
          "core":"testBasic_shard1_replica_t9",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10080_solr",
          "state":"active",
          "type":"TLOG",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node2":{
          "core":"testBasic_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10151_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node3":{
          "core":"testBasic_shard1_replica_n3",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10055_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node4":{
          "core":"testBasic_shard1_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10129_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node5":{
          "core":"testBasic_shard1_replica_n5",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10153_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-b332ffff",
      "state":"active"}}}
	at __randomizedtesting.SeedInfo.seed([D73038E4AA0C1079:7CCA25F175D09657]:0)
	at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:112)
	at org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testBasic(TestSimLargeCluster.java:239)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14310 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster
   [junit4]   2> 3594736 INFO  (SUITE-TestSimLargeCluster-seed#[D73038E4AA0C1079]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestSimLargeCluster_D73038E4AA0C1079-001/init-core-data-001
   [junit4]   2> 3594736 INFO  (SUITE-TestSimLargeCluster-seed#[D73038E4AA0C1079]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 3594737 INFO  (SUITE-TestSimLargeCluster-seed#[D73038E4AA0C1079]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 3594739 DEBUG (SUITE-TestSimLargeCluster-seed#[D73038E4AA0C1079]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10025_solr
   [junit4]   2> 3594740 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 3594740 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
   [junit4]   2> 3594740 INFO  (SUITE-TestSimLargeCluster-seed#[D73038E4AA0C1079]-worker) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3594740 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 3594740 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 3594740 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10031_solr, 127.0.0.1:10033_solr, 127.0.0.1:10025_solr, 127.0.0.1:10026_solr, 127.0.0.1:10029_solr, 127.0.0.1:10030_solr, 127.0.0.1:10032_solr, 127.0.0.1:10028_solr, 127.0.0.1:10027_solr]
   [junit4]   2> 3594743 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3594743 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
   [junit4]   2> 3594743 DEBUG (ScheduledTrigger-13259-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 81
   [junit4]   2> 3594748 INFO  (TEST-TestSimLargeCluster.testNodeLost-seed#[D73038E4AA0C1079]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLost
   [junit4]   2> 3594748 DEBUG (TEST-TestSimLargeCluster.testNodeLost-seed#[D73038E4AA0C1079]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 3594748 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 3594749 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 3594749 INFO  (TEST-TestSimLargeCluster.testNodeLost-seed#[D73038E4AA0C1079]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 3594749 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3594750 DEBUG (TEST-TestSimLargeCluster.testNodeLost-seed#[D73038E4AA0C1079]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 3594750 DEBUG (TEST-TestSimLargeCluster.testNodeLost-seed#[D73038E4AA0C1079]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 3594749 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_172]
   [junit4]   2> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_172]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 3594750 DEBUG (TEST-TestSimLargeCluster.testNodeLost-seed#[D73038E4AA0C1079]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 3594750 DEBUG (TEST-TestSimLargeCluster.testNodeLost-seed#[D73038E4AA0C1079]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 3594750 DEBUG (TEST-TestSimLargeCluster.testNodeLost-seed#[D73038E4AA0C1079]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 3594750 DEBUG (TEST-TestSimLargeCluster.testNodeLost-seed#[D73038E4AA0C1079]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 3594750 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 3594750 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 3594750 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 3594750 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 3594751 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10118_solr, 127.0.0.1:10049_solr, 127.0.0.1:10104_solr, 127.0.0.1:10065_solr, 127.0.0.1:10120_solr, 127.0.0.1:10066_solr, 127.0.0.1:10121_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10096_solr, 127.0.0.1:10119_solr, 127.0.0.1:10033_solr, 127.0.0.1:10105_solr, 127.0.0.1:10047_solr, 127.0.0.1:10102_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 127.0.0.1:10123_solr, 127.0.0.1:10071_solr, 127.0.0.1:10076_solr, 127.0.0.1:10107_solr, 127.0.0.1:10055_solr, 127.0.0.1:10124_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10108_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10117_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10114_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10111_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10112_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10113_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10030_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10115_solr, 127.0.0.1:10069_solr, 127.0.0.1:10110_solr, 127.0.0.1:10062_solr, 127.0.0.1:10116_solr, 127.0.0.1:10122_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10106_solr, 127.0.0.1:10100_solr, 127.0.0.1:10037_solr, 127.0.0.1:10109_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10103_solr, 127.0.0.1:10072_solr, 127.0.0.1:10056_solr]
   [junit4]   2> 3594751 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3594751 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 3594751 DEBUG (ScheduledTrigger-13264-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3594772 DEBUG (ScheduledTrigger-13264-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3594792 DEBUG (ScheduledTrigger-13264-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3594812 DEBUG (ScheduledTrigger-13264-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3594832 DEBUG (ScheduledTrigger-13264-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3594852 DEBUG (simCloudManagerPool-13263-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3594852 DEBUG (simCloudManagerPool-13263-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 3594853 DEBUG (ScheduledTrigger-13264-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 3594853 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 3594853 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3594853 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 3594854 DEBUG (simCloudManagerPool-13263-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3594854 DEBUG (simCloudManagerPool-13263-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 3594855 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 3594855 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 3594855 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3594855 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 3594876 DEBUG (simCloudManagerPool-13263-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider -- submit leader election for .system / shard1
   [junit4]   2> 3594876 DEBUG (simCloudManagerPool-13263-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider Running leader election for .system / shard1
   [junit4]   2> 3594876 DEBUG (simCloudManagerPool-13263-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1: core_node1
   [junit4]   2> 3594879 DEBUG (simCloudManagerPool-13263-thread-5) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3594879 DEBUG (simCloudManagerPool-13263-thread-5) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 3594879 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 3594879 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger3 - Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10118_solr, 127.0.0.1:10049_solr, 127.0.0.1:10104_solr, 127.0.0.1:10065_solr, 127.0.0.1:10120_solr, 127.0.0.1:10066_solr, 127.0.0.1:10121_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10096_solr, 127.0.0.1:10119_solr, 127.0.0.1:10033_solr, 127.0.0.1:10105_solr, 127.0.0.1:10047_solr, 127.0.0.1:10102_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 127.0.0.1:10123_solr, 127.0.0.1:10071_solr, 127.0.0.1:10076_solr, 127.0.0.1:10107_solr, 127.0.0.1:10055_solr, 127.0.0.1:10124_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10108_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10117_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10114_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10111_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10112_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10113_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10030_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10115_solr, 127.0.0.1:10069_solr, 127.0.0.1:10110_solr, 127.0.0.1:10062_solr, 127.0.0.1:10116_solr, 127.0.0.1:10122_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10106_solr, 127.0.0.1:10100_solr, 127.0.0.1:10037_solr, 127.0.0.1:10109_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10103_solr, 127.0.0.1:10072_solr, 127.0.0.1:10056_solr]
   [junit4]   2> 3594879 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3594879 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 3594879 DEBUG (ScheduledTrigger-13264-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger3 with currently live nodes: 100
   [junit4]   2> 3594881 DEBUG (simCloudManagerPool-13263-thread-6) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3594881 DEBUG (simCloudManagerPool-13263-thread-6) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 3594881 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 3594881 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3594881 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 3594899 DEBUG (ScheduledTrigger-13264-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger3 with currently live nodes: 100
   [junit4]   2> 3594902 DEBUG (simCloudManagerPool-13263-thread-8) [    ] o.a.s.c.a.s.SimClusterStateProvider -- submit leader election for testNodeLost / shard1
   [junit4]   2> 3594903 DEBUG (simCloudManagerPool-13263-thread-97) [    ] o.a.s.c.a.s.SimClusterStateProvider Running leader election for testNodeLost / shard1
   [junit4]   2> 3594903 DEBUG (simCloudManagerPool-13263-thread-97) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testNodeLost / shard1: core_node1
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard20
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard10
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard11
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard12
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard13
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard2
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard14
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard3
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard15
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard4
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard16
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard5
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard17
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard18
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard19
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- already has leader testNodeLost / shard1
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard6
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard7
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard8
   [junit4]   2> 3594914 DEBUG (simCloudManagerPool-13263-thread-9) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard9
   [junit4]   2> 3594920 DEBUG (ScheduledTrigger-13264-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger3 with currently live nodes: 100
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard20
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard10
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard11
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard12
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard13
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard2
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard14
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard3
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard15
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard4
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard16
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard5
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard17
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard18
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard19
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- already has leader testNodeLost / shard1
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard6
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard7
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard8
   [junit4]   2> 3594926 DEBUG (simCloudManagerPool-13263-thread-10) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard9
   [junit4]   2> 3594936 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard20
   [junit4]   2> 3594936 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard10
   [junit4]   2> 3594936 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard11
   [junit4]   2> 3594936 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard12
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard13
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard2
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard14
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard3
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard15
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard4
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard16
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard5
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard17
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard18
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard19
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- already has leader testNodeLost / shard1
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard6
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard7
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard8
   [junit4]   2> 3594937 DEBUG (simCloudManagerPool-13263-thread-11) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard9
   [junit4]   2> 3594940 DEBUG (ScheduledTrigger-13264-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger3 with currently live nodes: 100
   [junit4]   2> 3594947 DEBUG (simCloudManagerPool-13263-thread-12) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no replicas in testNodeLost / shard20
   [junit4]   2> 3594947 DEBUG (simCloudManagerPool-13263-thread-12) [    ] o.a.s.c.a.s.SimClusterStateProvid

[...truncated too long message...]


-clover.classpath:

-clover.setup:

clover:

compile-core:

-clover.disable:

-clover.load:

-clover.classpath:

-clover.setup:

clover:

common.compile-core:

compile-core:

common.compile-test:
    [mkdir] Created dir: /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/classes/test
    [javac] Compiling 932 source files to /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/classes/test
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] Creating empty /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/classes/test/org/apache/solr/cloud/autoscaling/sim/package-info.class
     [copy] Copying 1 file to /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/classes/test

common-solr.compile-test:

compile-test:

BUILD SUCCESSFUL
Total time: 54 seconds
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.TestSimLargeCluster" -Dtests.showOutput=onerror  -Dtests.seed=D73038E4AA0C1079 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=id -Dtests.timezone=Africa/Maseru -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
Buildfile: /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/core/build.xml

-clover.disable:

ivy-configure:
[ivy:configure] :: Apache Ivy 2.4.0 - 20141213170938 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/lucene/top-level-ivy-settings.xml

install-junit4-taskdef:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

resolve-groovy:
[ivy:cachepath] :: resolving dependencies :: org.codehaus.groovy#groovy-all-caller;working
[ivy:cachepath] 	confs: [default]
[ivy:cachepath] 	found org.codehaus.groovy#groovy-all;2.4.15 in public
[ivy:cachepath] :: resolution report :: resolve 17ms :: artifacts dl 1ms
	---------------------------------------------------------------------
	|                  |            modules            ||   artifacts   |
	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
	---------------------------------------------------------------------
	|      default     |   1   |   0   |   0   |   0   ||   1   |   0   |
	---------------------------------------------------------------------

-init-totals:

-test:
    [mkdir] Created dir: /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test
[junit4:pickseed] Seed property 'tests.seed' already defined: D73038E4AA0C1079
    [mkdir] Created dir: /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/temp
   [junit4] <JUnit4> says olá! Master seed: D73038E4AA0C1079
   [junit4] Executing 5 suites with 3 JVMs.
   [junit4] 
   [junit4] Started J1 PID(12073@serv1.sd-datasolutions.de).
   [junit4] Started J2 PID(12075@serv1.sd-datasolutions.de).
   [junit4] Started J0 PID(12074@serv1.sd-datasolutions.de).
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster
   [junit4] OK      17.2s J2 | TestSimLargeCluster.testNodeLost
   [junit4] OK      7.18s J2 | TestSimLargeCluster.testAddNode
   [junit4] OK      1.88s J2 | TestSimLargeCluster.testSearchRate
   [junit4] OK      5.98s J2 | TestSimLargeCluster.testBasic
   [junit4] Completed [1/5] on J2 in 34.58s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster
   [junit4] OK      17.5s J0 | TestSimLargeCluster.testNodeLost
   [junit4] OK      7.24s J0 | TestSimLargeCluster.testAddNode
   [junit4] OK      1.89s J0 | TestSimLargeCluster.testSearchRate
   [junit4] OK      6.01s J0 | TestSimLargeCluster.testBasic
   [junit4] Completed [2/5] on J0 in 34.91s, 4 tests
   [junit4] 
   [junit4] Duplicate suite name used with XML reports: org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster. This may confuse tools that process XML reports. Set 'ignoreDuplicateSuites' to true to skip this message.
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster
   [junit4] OK      17.1s J1 | TestSimLargeCluster.testNodeLost
   [junit4] OK      6.98s J1 | TestSimLargeCluster.testAddNode
   [junit4] OK      1.82s J1 | TestSimLargeCluster.testSearchRate
   [junit4] OK      6.92s J1 | TestSimLargeCluster.testBasic
   [junit4] Completed [3/5] on J1 in 35.06s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster
   [junit4] OK      16.4s J2 | TestSimLargeCluster.testNodeLost
   [junit4] OK      6.81s J2 | TestSimLargeCluster.testAddNode
   [junit4] OK      1.65s J2 | TestSimLargeCluster.testSearchRate
   [junit4] OK      6.69s J2 | TestSimLargeCluster.testBasic
   [junit4] Completed [4/5] on J2 in 31.60s, 4 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster
   [junit4] OK      16.3s J0 | TestSimLargeCluster.testNodeLost
   [junit4] OK      6.91s J0 | TestSimLargeCluster.testAddNode
   [junit4] OK      1.61s J0 | TestSimLargeCluster.testSearchRate
   [junit4] OK      7.10s J0 | TestSimLargeCluster.testBasic
   [junit4] Completed [5/5] on J0 in 32.28s, 4 tests
   [junit4] 
   [junit4] JVM J0:     0.68 ..    68.76 =    68.08s
   [junit4] JVM J1:     0.42 ..    36.49 =    36.07s
   [junit4] JVM J2:     0.67 ..    67.78 =    67.10s
   [junit4] Execution time total: 1 minute 8 seconds
   [junit4] Tests summary: 5 suites, 20 tests
   [junit4] Could not remove temporary path: /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1 (java.nio.file.DirectoryNotEmptyException: Remaining files: [/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J1/temp])
   [junit4] Could not remove temporary path: /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0 (java.nio.file.DirectoryNotEmptyException: Remaining files: [/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp])
   [junit4] Could not remove temporary path: /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J2 (java.nio.file.DirectoryNotEmptyException: Remaining files: [/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J2/temp])
     [echo] 5 slowest tests:
[junit4:tophints] 1347.84s | org.apache.solr.cloud.BasicDistributedZkTest
[junit4:tophints] 1063.94s | org.apache.solr.cloud.cdcr.CdcrOpsAndBoundariesTest
[junit4:tophints] 952.96s | org.apache.solr.cloud.cdcr.CdcrWithNodesRestartsTest
[junit4:tophints] 512.15s | org.apache.solr.cloud.api.collections.ShardSplitTest
[junit4:tophints] 208.74s | org.apache.solr.update.TestInPlaceUpdatesDistrib

-check-totals:

test-nocompile:

BUILD SUCCESSFUL
Total time: 1 minute 10 seconds
[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster
[repro] Exiting with code 0
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
java.lang.InterruptedException: no matches found within 10000
	at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2845)
	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2724)
	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2705)
	at hudson.FilePath.act(FilePath.java:1076)
	at hudson.FilePath.act(FilePath.java:1059)
	at hudson.FilePath.validateAntFileMask(FilePath.java:2703)
	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
	at hudson.model.Build$BuildExecution.post2(Build.java:186)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
	at hudson.model.Run.execute(Run.java:1840)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/*_pid*.log". Configuration error?
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=70f001918ebaacbc83b1a7b2c0209b6f37a80813, workspace=/var/lib/jenkins/workspace/Lucene-Solr-BadApples-master-Linux
[WARNINGS] Computing warning deltas based on reference build #89
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2

[JENKINS] Lucene-Solr-BadApples-master-Linux (64bit/jdk-10.0.1) - Build # 89 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-BadApples-master-Linux/89/
Java: 64bit/jdk-10.0.1 -XX:-UseCompressedOops -XX:+UseParallelGC

12 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates

Error Message:


Stack Trace:
java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59]:0)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.createMiniSolrCloudCluster(TestStressCloudBlindAtomicUpdates.java:138)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates

Error Message:


Stack Trace:
java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59]:0)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.afterClass(TestStressCloudBlindAtomicUpdates.java:158)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [InternalHttpClient] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:268)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:255)  at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:200)  at org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:951)  at org.apache.solr.SolrTestCaseJ4.getHttpSolrClient(SolrTestCaseJ4.java:2572)  at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.createMiniSolrCloudCluster(TestStressCloudBlindAtomicUpdates.java:138)  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  at java.base/java.lang.reflect.Method.invoke(Method.java:564)  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)  at java.base/java.lang.Thread.run(Thread.java:844)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [InternalHttpClient]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:268)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:200)
	at org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:951)
	at org.apache.solr.SolrTestCaseJ4.getHttpSolrClient(SolrTestCaseJ4.java:2572)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.createMiniSolrCloudCluster(TestStressCloudBlindAtomicUpdates.java:138)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:304)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates:     1) Thread[id=531, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestStressCloudBlindAtomicUpdates]         at java.base@10.0.1/java.lang.Thread.sleep(Native Method)         at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates: 
   1) Thread[id=531, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestStressCloudBlindAtomicUpdates]
        at java.base@10.0.1/java.lang.Thread.sleep(Native Method)
        at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates

Error Message:


Stack Trace:
java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59]:0)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.createMiniSolrCloudCluster(TestStressCloudBlindAtomicUpdates.java:138)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates

Error Message:


Stack Trace:
java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59]:0)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.afterClass(TestStressCloudBlindAtomicUpdates.java:158)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [InternalHttpClient] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:268)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:255)  at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:200)  at org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:951)  at org.apache.solr.SolrTestCaseJ4.getHttpSolrClient(SolrTestCaseJ4.java:2572)  at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.createMiniSolrCloudCluster(TestStressCloudBlindAtomicUpdates.java:138)  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  at java.base/java.lang.reflect.Method.invoke(Method.java:564)  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)  at java.base/java.lang.Thread.run(Thread.java:844)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [InternalHttpClient]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:268)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:200)
	at org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:951)
	at org.apache.solr.SolrTestCaseJ4.getHttpSolrClient(SolrTestCaseJ4.java:2572)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.createMiniSolrCloudCluster(TestStressCloudBlindAtomicUpdates.java:138)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:304)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates:     1) Thread[id=535, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestStressCloudBlindAtomicUpdates]         at java.base@10.0.1/java.lang.Thread.sleep(Native Method)         at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates: 
   1) Thread[id=535, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestStressCloudBlindAtomicUpdates]
        at java.base@10.0.1/java.lang.Thread.sleep(Native Method)
        at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59]:0)


FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv

Error Message:
Some docs had errors -- check logs expected:<0> but was:<20>

Stack Trace:
java.lang.AssertionError: Some docs had errors -- check logs expected:<0> but was:<20>
	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59:2863F6BB1F720048]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:343)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv(TestStressCloudBlindAtomicUpdates.java:198)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_stored_idx

Error Message:
expected:<234> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<234> but was:<0>
	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59:E163904808D7F77]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:273)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_stored_idx(TestStressCloudBlindAtomicUpdates.java:242)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_stored_idx

Error Message:
expected:<304> but was:<147>

Stack Trace:
java.lang.AssertionError: expected:<304> but was:<147>
	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59:1450C5DDEA59990E]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:273)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_stored_idx(TestStressCloudBlindAtomicUpdates.java:221)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv

Error Message:
Some docs had errors -- check logs expected:<0> but was:<12>

Stack Trace:
java.lang.AssertionError: Some docs had errors -- check logs expected:<0> but was:<12>
	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59:2863F6BB1F720048]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:343)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv(TestStressCloudBlindAtomicUpdates.java:198)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 12895 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_1E7794FD952F3A59-001/init-core-data-001
   [junit4]   2> 437730 WARN  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[1E7794FD952F3A59]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=8 numCloses=8
   [junit4]   2> 437730 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[1E7794FD952F3A59]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 437731 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[1E7794FD952F3A59]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl="SSL overhead seems to cause OutOfMemory when stress testing")
   [junit4]   2> 437731 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[1E7794FD952F3A59]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 437732 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[1E7794FD952F3A59]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_1E7794FD952F3A59-001/tempDir-001
   [junit4]   2> 437732 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[1E7794FD952F3A59]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 437737 INFO  (Thread-1244) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 437737 INFO  (Thread-1244) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 437749 ERROR (Thread-1244) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 437837 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[1E7794FD952F3A59]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:38507
   [junit4]   2> 437849 INFO  (zkConnectionManagerCallback-4196-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 437851 INFO  (jetty-launcher-4193-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 10.0.1+10
   [junit4]   2> 437854 INFO  (jetty-launcher-4193-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 10.0.1+10
   [junit4]   2> 437860 INFO  (jetty-launcher-4193-thread-4) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 10.0.1+10
   [junit4]   2> 437860 INFO  (jetty-launcher-4193-thread-3) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 10.0.1+10
   [junit4]   2> 437878 INFO  (jetty-launcher-4193-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 437878 INFO  (jetty-launcher-4193-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 437878 INFO  (jetty-launcher-4193-thread-4) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 437878 INFO  (jetty-launcher-4193-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 437878 INFO  (jetty-launcher-4193-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 437878 INFO  (jetty-launcher-4193-thread-3) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 437881 INFO  (jetty-launcher-4193-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 437881 INFO  (jetty-launcher-4193-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 437881 INFO  (jetty-launcher-4193-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 437886 INFO  (jetty-launcher-4193-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2470ce0{/solr,null,AVAILABLE}
   [junit4]   2> 437886 INFO  (jetty-launcher-4193-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@aeca60{/solr,null,AVAILABLE}
   [junit4]   2> 437888 INFO  (jetty-launcher-4193-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@70161d1c{HTTP/1.1,[http/1.1]}{127.0.0.1:44681}
   [junit4]   2> 437888 INFO  (jetty-launcher-4193-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@49b50f99{HTTP/1.1,[http/1.1]}{127.0.0.1:40949}
   [junit4]   2> 437888 INFO  (jetty-launcher-4193-thread-4) [    ] o.e.j.s.Server Started @437927ms
   [junit4]   2> 437888 INFO  (jetty-launcher-4193-thread-3) [    ] o.e.j.s.Server Started @437927ms
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 437888 INFO  (jetty-launcher-4193-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44681}
   [junit4]   2> 437888 INFO  (jetty-launcher-4193-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40949}
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@29eeaf85{/solr,null,AVAILABLE}
   [junit4]   2> 437889 ERROR (jetty-launcher-4193-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-4) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 437889 ERROR (jetty-launcher-4193-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-08T04:03:38.787328Z
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-08T04:03:38.787328Z
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2c92e5ff{HTTP/1.1,[http/1.1]}{127.0.0.1:34199}
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-2) [    ] o.e.j.s.Server Started @437927ms
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34199}
   [junit4]   2> 437889 ERROR (jetty-launcher-4193-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-08T04:03:38.787550Z
   [junit4]   2> 437889 INFO  (jetty-launcher-4193-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3e200b70{/solr,null,AVAILABLE}
   [junit4]   2> 437890 INFO  (jetty-launcher-4193-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@574674f1{HTTP/1.1,[http/1.1]}{127.0.0.1:43989}
   [junit4]   2> 437890 INFO  (jetty-launcher-4193-thread-1) [    ] o.e.j.s.Server Started @437928ms
   [junit4]   2> 437890 INFO  (jetty-launcher-4193-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43989}
   [junit4]   2> 437890 ERROR (jetty-launcher-4193-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 437890 INFO  (jetty-launcher-4193-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 437890 INFO  (jetty-launcher-4193-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 437890 INFO  (jetty-launcher-4193-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 437890 INFO  (jetty-launcher-4193-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 437890 INFO  (jetty-launcher-4193-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-09-08T04:03:38.788271Z
   [junit4]   2> 437901 INFO  (zkConnectionManagerCallback-4204-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 437901 INFO  (zkConnectionManagerCallback-4199-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 437901 INFO  (jetty-launcher-4193-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 437901 INFO  (jetty-launcher-4193-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 437901 INFO  (zkConnectionManagerCallback-4200-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 437901 INFO  (jetty-launcher-4193-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 437908 INFO  (zkConnectionManagerCallback-4202-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 437909 INFO  (jetty-launcher-4193-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 437919 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10075b993c90003, likely client has closed socket
   [junit4]   2> 437919 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10075b993c90002, likely client has closed socket
   [junit4]   2> 438046 INFO  (jetty-launcher-4193-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38507/solr
   [junit4]   2> 438060 INFO  (zkConnectionManagerCallback-4208-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 438064 INFO  (jetty-launcher-4193-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38507/solr
   [junit4]   2> 438073 INFO  (zkConnectionManagerCallback-4211-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 438077 INFO  (zkConnectionManagerCallback-4214-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 438086 INFO  (jetty-launcher-4193-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38507/solr
   [junit4]   2> 438089 INFO  (zkConnectionManagerCallback-4217-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 438093 INFO  (jetty-launcher-4193-thread-1) [n:127.0.0.1:43989_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:43989_solr
   [junit4]   2> 438094 INFO  (jetty-launcher-4193-thread-1) [n:127.0.0.1:43989_solr    ] o.a.s.c.Overseer Overseer (id=72187033946750984-127.0.0.1:43989_solr-n_0000000000) starting
   [junit4]   2> 438096 INFO  (zkConnectionManagerCallback-4222-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 438112 INFO  (zkConnectionManagerCallback-4227-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 438113 INFO  (zkConnectionManagerCallback-4231-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 438114 INFO  (jetty-launcher-4193-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38507/solr
   [junit4]   2> 438115 INFO  (jetty-launcher-4193-thread-1) [n:127.0.0.1:43989_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38507/solr ready
   [junit4]   2> 438118 INFO  (jetty-launcher-4193-thread-2) [n:127.0.0.1:34199_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34199_solr
   [junit4]   2> 438122 INFO  (jetty-launcher-4193-thread-4) [n:127.0.0.1:44681_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 438124 INFO  (jetty-launcher-4193-thread-4) [n:127.0.0.1:44681_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 438124 INFO  (jetty-launcher-4193-thread-4) [n:127.0.0.1:44681_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44681_solr
   [junit4]   2> 438149 INFO  (zkConnectionManagerCallback-4235-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 438149 INFO  (zkCallback-4210-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 438149 INFO  (zkCallback-4230-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 438150 INFO  (zkCallback-4226-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 438150 INFO  (zkCallback-4216-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 438161 INFO  (OverseerStateUpdate-72187033946750984-127.0.0.1:43989_solr-n_0000000000) [n:127.0.0.1:43989_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:43989_solr
   [junit4]   2> 438162 INFO  (zkConnectionManagerCallback-4247-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 438163 INFO  (zkConnectionManagerCallback-4243-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 438163 INFO  (jetty-launcher-4193-thread-1) [n:127.0.0.1:43989_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 438163 INFO  (jetty-launcher-4193-thread-1) [n:127.0.0.1:43989_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43989_solr
   [junit4]   2> 438163 INFO  (jetty-launcher-4193-thread-4) [n:127.0.0.1:44681_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 438163 INFO  (zkConnectionManagerCallback-4249-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 438164 INFO  (jetty-launcher-4193-thread-4) [n:127.0.0.1:44681_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38507/solr ready
   [junit4]   2> 438164 INFO  (zkCallback-4210-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 438164 INFO  (zkCallback-4216-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 438164 INFO  (zkCallback-4230-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 438164 INFO  (jetty-launcher-4193-thread-2) [n:127.0.0.1:34199_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 438164 INFO  (jetty-launcher-4193-thread-2) [n:127.0.0.1:34199_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38507/solr ready
   [junit4]   2> 438165 INFO  (zkCallback-4226-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 438167 INFO  (jetty-launcher-4193-thread-3) [n:127.0.0.1:40949_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 438175 INFO  (jetty-launcher-4193-thread-3) [n:127.0.0.1:40949_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 438175 INFO  (jetty-launcher-4193-thread-3) [n:127.0.0.1:40949_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40949_solr
   [junit4]   2> 438175 INFO  (zkCallback-4216-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 438177 INFO  (zkCallback-4230-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 438177 INFO  (zkCallback-4210-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 438177 INFO  (zkCallback-4226-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 438179 INFO  (jetty-launcher-4193-thread-2) [n:127.0.0.1:34199_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 438179 INFO  (jetty-launcher-4193-thread-1) [n:127.0.0.1:43989_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 438190 INFO  (zkCallback-4246-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 438190 INFO  (zkCallback-4242-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 438190 INFO  (zkCallback-4248-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 438190 INFO  (zkCallback-4246-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 438190 INFO  (zkConnectionManagerCallback-4259-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 438190 INFO  (jetty-launcher-4193-thread-4) [n:127.0.0.1:44681_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 438190 INFO  (jetty-launcher-4193-thread-3) [n:127.0.0.1:40949_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 438192 INFO  (jetty-launcher-4193-thread-3) [n:127.0.0.1:40949_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38507/solr ready
   [junit4]   2> 438193 INFO  (jetty-launcher-4193-thread-3) [n:127.0.0.1:40949_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 438194 INFO  (jetty-launcher-4193-thread-2) [n:127.0.0.1:34199_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34199.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 438203 INFO  (jetty-launcher-4193-thread-2) [n:127.0.0.1:34199_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34199.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 438203 INFO  (jetty-launcher-4193-thread-2) [n:127.0.0.1:34199_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34199.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 438204 INFO  (jetty-launcher-4193-thread-2) [n:127.0.0.1:34199_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_1E7794FD952F3A59-001/tempDir-001/node2/.
   [junit4]   2> 438205 INFO  (jetty-launcher-4193-thread-1) [n:127.0.0.1:43989_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43989.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 438207 INFO  (jetty-launcher-4193-thread-4) [n:127.0.0.1:44681_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44681.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 438210 INFO  (jetty-launcher-4193-thread-1) [n:127.0.0.1:43989_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43989.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 438210 INFO  (jetty-launcher-4193-thread-1) [n:127.0.0.1:43989_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43989.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 438210 INFO  (jetty-launcher-4193-thread-1) [n:127.0.0.1:43989_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_1E7794FD952F3A59-001/tempDir-001/node1/.
   [junit4]   2> 438211 INFO  (jetty-launcher-4193-thread-3) [n:127.0.0.1:40949_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40949.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 438214 INFO  (jetty-launcher-4193-thread-4) [n:127.0.0.1:44681_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44681.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 438214 INFO  (jetty-launcher-4193-thread-4) [n:127.0.0.1:44681_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44681.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 438215 INFO  (jetty-launcher-4193-thread-4) [n:127.0.0.1:44681_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_1E7794FD952F3A59-001/tempDir-001/node4/.
   [junit4]   2> 438219 INFO  (jetty-launcher-4193-thread-3) [n:127.0.0.1:40949_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40949.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 438219 INFO  (jetty-launcher-4193-thread-3) [n:127.0.0.1:40949_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40949.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 438219 INFO  (jetty-launcher-4193-thread-3) [n:127.0.0.1:40949_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_1E7794FD952F3A59-001/tempDir-001/node3/.
   [junit4]   2> 438308 INFO  (zkConnectionManagerCallback-4262-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 438309 INFO  (zkConnectionManagerCallback-4267-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 438310 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[1E7794FD952F3A59]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 438310 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[1E7794FD952F3A59]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38507/solr ready
   [junit4]   2> 438603 INFO  (qtp682839801-8283) [n:127.0.0.1:43989_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params property.schema=schema-minimal-atomic-stress.xml&property.config=solrconfig-tlog.xml&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&name=test_col&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 438605 INFO  (OverseerThreadFactory-2634-thread-1-processing-n:127.0.0.1:43989_solr) [n:127.0.0.1:43989_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection test_col
   [junit4]   2> 438723 INFO  (OverseerStateUpdate-72187033946750984-127.0.0.1:43989_solr-n_0000000000) [n:127.0.0.1:43989_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"test_col",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"test_col_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:43989/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 438725 INFO  (OverseerStateUpdate-72187033946750984-127.0.0.1:43989_solr-n_0000000000) [n:127.0.0.1:43989_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"test_col",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"test_col_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:44681/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 438733 INFO  (OverseerStateUpdate-72187033946750984-127.0.0.1:43989_solr-n_0000000000) [n:127.0.0.1:43989_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"test_col",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"test_col_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:34199/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 438735 INFO  (OverseerStateUpdate-72187033946750984-127.0.0.1:43989_solr-n_0000000000) [n:127.0.0.1:43989_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"test_col",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"test_col_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:40949/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 438983 INFO  (qtp682839801-8257) [n:127.0.0.1:43989_solr    x:test_col_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.schema=schema-minimal-atomic-stress.xml&property.config=solrconfig-tlog.xml&coreNodeName=core_node3&name=test_col_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 438984 INFO  (qtp679845575-8282) [n:127.0.0.1:44681_solr    x:test_col_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.schema=schema-minimal-atomic-stress.xml&property.config=solrconfig-tlog.xml&coreNodeName=core_node5&name=test_col_shard1_replica_n2&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 438994 INFO  (qtp1245039655-8281) [n:127.0.0.1:40949_solr    x:test_col_shard2_replica_n6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.schema=schema-minimal-atomic-stress.xml&property.config=solrconfig-tlog.xml&coreNodeName=core_node8&name=test_col_shard2_replica_n6&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 439000 INFO  (qtp973848173-8280) [n:127.0.0.1:34199_solr    x:test_col_shard2_replica_n4] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.schema=schema-minimal-atomic-stress.xml&property.config=solrconfig-tlog.xml&coreNodeName=core_node7&name=test_col_shard2_replica_n4&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 439000 INFO  (qtp973848173-8280) [n:127.0.0.1:34199_solr    x:test_col_shard2_replica_n4] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 440027 INFO  (qtp679845575-8282) [n:127.0.0.1:44681_solr c:test_col s:shard1 r:core_node5 x:test_col_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 440030 INFO  (qtp973848173-8280) [n:127.0.0.1:34199_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 440055 INFO  (qtp682839801-8257) [n:127.0.0.1:43989_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 440056 INFO  (qtp1245039655-8281) [n:127.0.0.1:40949_solr c:test_col s:shard2 r:core_node8 x:test_col_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 440061 INFO  (qtp679845575-8282) [n:127.0.0.1:44681_solr c:test_col s:shard1 r:core_node5 x:test_col_shard1_replica_n2] o.a.s.s.IndexSchema [test_col_shard1_replica_n2] Schema name=minimal-atomic-stress
   [junit4]   2> 440065 INFO  (qtp679845575-8282) [n:127.0.0.1:44681_solr c:test_col s:shard1 r:core_node5 x:test_col_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 440065 INFO  (qtp679845575-8282) [n:127.0.0.1:44681_solr c:test_col s:shard1 r:core_node5 x:test_col_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard1_replica_n2' using configuration from collection test_col, trusted=true
   [junit4]   2> 440066 INFO  (qtp679845575-8282) [n:127.0.0.1:44681_solr c:test_col s:shard1 r:core_node5 x:test_col_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44681.solr.core.test_col.shard1.replica_n2' (registry 'solr.core.test_col.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 440067 INFO  (qtp679845575-8282) [n:127.0.0.1:44681_solr c:test_col s:shard1 r:core_node5 x:test_col_shard1_replica_n2] o.a.s.c.SolrCore [[test_col_shard1_replica_n2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_1E7794FD952F3A59-001/tempDir-001/node4/test_col_shard1_replica_n2], dataDir=[/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_1E7794FD952F3A59-001/tempDir-001/node4/./test_col_shard1_replica_n2/data/]
   [junit4]   2> 440072 INFO  (qtp679845575-8282) [n:127.0.0.1:44681_solr c:test_col s:shard1 r:core_node5 x:test_col_shard1_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=36, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 440074 INFO  (qtp682839801-8257) [n:127.0.0.1:43989_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.s.IndexSchema [test_col_shard1_replica_n1] Schema name=minimal-atomic-stress
   [junit4]   2> 440075 INFO  (qtp973848173-8280) [n:127.0.0.1:34199_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.s.IndexSchema [test_col_shard2_replica_n4] Schema name=minimal-atomic-stress
   [junit4]   2> 440077 INFO  (qtp973848173-8280) [n:127.0.0.1:34199_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 440077 INFO  (qtp973848173-8280) [n:127.0.0.1:34199_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard2_replica_n4' using configuration from collection test_col, trusted=true
   [junit4]   2> 440077 INFO  (qtp973848173-8280) [n:127.0.0.1:34199_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34199.solr.core.test_col.shard2.replica_n4' (registry 'solr.core.test_col.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 440077 INFO  (qtp1245039655-8281) [n:127.0.0.1:40949_solr c:test_col s:shard2 r:core_node8 x:test_col_shard2_replica_n6] o.a.s.s.IndexSchema [test_col_shard2_replica_n6] Schema name=minimal-atomic-stress
   [junit4]   2> 440079 INFO  (qtp682839801-8257) [n:127.0.0.1:43989_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 440079 INFO  (qtp682839801-8257) [n:127.0.0.1:43989_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard1_replica_n1' using configuration from collection test_col, trusted=true
   [junit4]   2> 440079 INFO  (qtp1245039655-8281) [n:127.0.0.1:40949_solr c:test_col s:shard2 r:core_node8 x:test_col_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 440079 INFO  (qtp1245039655-8281) [n:127.0.0.1:40949_solr c:test_col s:shard2 r:core_node8 x:test_col_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard2_replica_n6' using configuration from collection test_col, trusted=true
   [junit4]   2> 440079 INFO  (qtp1245039655-8281) [n:127.0.0.1:40949_solr c:test_col s:shard2 r:core_node8 x:test_col_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40949.solr.core.test_col.shard2.replica_n6' (registry 'solr.core.test_col.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 440079 INFO  (qtp682839801-8257) [n:127.0.0.1:43989_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43989.solr.core.test_col.shard1.replica_n1' (registry 'solr.core.test_col.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c5ad161
   [junit4]   2> 440080 INFO  (qtp1245039655-8281) [n:127.0.0.1:40949_solr c:test_col s:shard2 r:core_node8 x:test_col_shard2_replica_n6] o.a.s.c.SolrCore [[test_col_shard2_replica_n6] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_1E7794FD952F3A59-001/tempDir-001/node3/test_col_shard2_replica_n6], dataDir=[/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_1E7794FD952F3A59-001/tempDir-001/node3/./test_col_shard2_replica_n6/data/]
   [junit4]   2> 440080 INFO  (qtp682839801-8257) [n:127.0.0.1:43989_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.c.SolrCore [[test_col_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_1E7794FD952F3A59-001/tempDir-001/node1/test_col_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_1E7794FD952F3A59-001/tempDir-001/node1/./test_col_shard1_replica_n1/data/]
   [junit4]   2> 440080 INFO  (qtp973848173-8280) [n:127.0.0.1:34199_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.c.SolrCore [[test_col_shard2_replica_n4] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_1E7794FD952F3A59-001/tempDir-001/node2/test_col_shard2_replica_n4], dataDir=[/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_1E7794FD952F3A59-001/tempDir-001/node2/./test_col_shard2_replica_n4/data/]
   [junit4]   2> 440084 INFO  (qtp1245039655-8281) [n:127.0.0.1:40949_solr c:test_col s:shard2 r:core_node8 x:test_col_shard2_replica_n6] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=36, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 440084 INFO  (qtp682839801-8257) [n:127.0.0.1:43989_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=36, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 440097 WARN  (qtp679845575-8282) [n:127.0.0.1:44681_solr c:test_col s:shard1 r:core_node5 x:test_col_shard1_replica_n2] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 440098 INFO  (qtp973848173-8280) [n:127.0.0.1:34199_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=36, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 440098 WARN  (qtp1245039655-8281) [n:127.0.0.1:40949_solr c:test_col s:shard2 r:core_node8 x:test_col_shard2_replica_n6] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 440098 WARN  (qtp682839801-8257) [n:127.0.0.1:43989_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 440132 WARN  (qtp973848173-8280) [n:127.0.0.1:34199_solr c:test_col s:shard2 r:core_node7 x:test

[...truncated too long message...]

  [junit4]   2>     app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:627)
   [junit4]   2> 
   [junit4]   2> main:
   [junit4]   2>     java.base@10.0.1/java.lang.Object.wait(Native Method)
   [junit4]   2>     java.base@10.0.1/java.lang.Thread.join(Thread.java:1353)
   [junit4]   2>     java.base@10.0.1/java.lang.Thread.join(Thread.java:1427)
   [junit4]   2>     app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:637)
   [junit4]   2>     app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:494)
   [junit4]   2>     app//com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:269)
   [junit4]   2>     app//com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:394)
   [junit4]   2>     app//com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2> 
   [junit4]   2> 99129 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[1E7794FD952F3A59]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Sep 08, 2018 5:16:17 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Sep 08, 2018 5:17:38 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates: 
   [junit4]   2>    1) Thread[id=535, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestStressCloudBlindAtomicUpdates]
   [junit4]   2>         at java.base@10.0.1/java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]   2>         at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> Sep 08, 2018 5:17:38 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=535, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestStressCloudBlindAtomicUpdates]
   [junit4]   2> Sep 08, 2018 5:17:38 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@10fce655), locale=sr-Cyrl-ME, timezone=Pacific/Truk
   [junit4]   2> NOTE: Linux 4.15.0-32-generic amd64/Oracle Corporation 10.0.1 (64-bit)/cpus=8,threads=1,free=213263432,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestStressCloudBlindAtomicUpdates, TestStressCloudBlindAtomicUpdates]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestStressCloudBlindAtomicUpdates -Dtests.seed=1E7794FD952F3A59 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=sr-Cyrl-ME -Dtests.timezone=Pacific/Truk -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | TestStressCloudBlindAtomicUpdates (suite) <<<
   [junit4]    > Throwable #1: java.lang.NullPointerException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59]:0)
   [junit4]    > 	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.createMiniSolrCloudCluster(TestStressCloudBlindAtomicUpdates.java:138)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:844)Throwable #2: java.lang.NullPointerException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59]:0)
   [junit4]    > 	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.afterClass(TestStressCloudBlindAtomicUpdates.java:158)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:844)Throwable #3: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [InternalHttpClient]
   [junit4]    > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
   [junit4]    > 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:268)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:255)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:200)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:951)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.getHttpSolrClient(SolrTestCaseJ4.java:2572)
   [junit4]    > 	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.createMiniSolrCloudCluster(TestStressCloudBlindAtomicUpdates.java:138)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:304)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:844)Throwable #4: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates: 
   [junit4]    >    1) Thread[id=535, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestStressCloudBlindAtomicUpdates]
   [junit4]    >         at java.base@10.0.1/java.lang.Thread.sleep(Native Method)
   [junit4]    >         at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1E7794FD952F3A59]:0)
   [junit4] Completed [5/5 (4!)] on J0 in 109.97s, 0 tests, 1 failure, 3 errors <<< FAILURES!

[...truncated 19 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/lucene/common-build.xml:1567: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-BadApples-master-Linux/lucene/common-build.xml:1092: There were test failures: 5 suites, 15 tests, 8 suite-level errors, 7 failures [seed: 1E7794FD952F3A59]

Total time: 3 minutes 2 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   4/5 failed: org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=8f498920bd4f2d0059031251409c812bab55404d, workspace=/var/lib/jenkins/workspace/Lucene-Solr-BadApples-master-Linux
[WARNINGS] Computing warning deltas based on reference build #88
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2