You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2018/05/21 22:29:01 UTC

[JENKINS] Lucene-Solr-BadApples-Tests-7.x - Build # 64 - Still Unstable

Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/64/

3 tests failed.
FAILED:  org.apache.solr.cloud.TestAuthenticationFramework.testBasics

Error Message:
Error from server at http://127.0.0.1:38720/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason: <pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.10.v20180503</a><hr/>  </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:38720/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.10.v20180503</a><hr/>

</body>
</html>

	at __randomizedtesting.SeedInfo.seed([E67EF89C67B9ADC0:DBA656B05F57F3B0]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1015)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
	at org.apache.solr.cloud.TestAuthenticationFramework.collectionCreateSearchDeleteTwice(TestAuthenticationFramework.java:127)
	at org.apache.solr.cloud.TestAuthenticationFramework.testBasics(TestAuthenticationFramework.java:75)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:38720/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.10.v20180503</a><hr/>

</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:192)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue.testDistributedQueue

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([E67EF89C67B9ADC0]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([E67EF89C67B9ADC0]:0)




Build Log:
[...truncated 14290 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestAuthenticationFramework
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_E67EF89C67B9ADC0-001/init-core-data-001
   [junit4]   2> 2861823 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[E67EF89C67B9ADC0]) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_E67EF89C67B9ADC0-001/tempDir-001
   [junit4]   2> 2861824 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[E67EF89C67B9ADC0]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2861825 INFO  (Thread-4794) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2861825 INFO  (Thread-4794) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2861845 ERROR (Thread-4794) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2861925 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[E67EF89C67B9ADC0]) [    ] o.a.s.c.ZkTestServer start zk server on port:51983
   [junit4]   2> 2861958 INFO  (zkConnectionManagerCallback-5529-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2861975 INFO  (jetty-launcher-5526-thread-1) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
   [junit4]   2> 2861993 INFO  (jetty-launcher-5526-thread-2) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
   [junit4]   2> 2862080 INFO  (jetty-launcher-5526-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2862080 INFO  (jetty-launcher-5526-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2862080 INFO  (jetty-launcher-5526-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2862086 INFO  (jetty-launcher-5526-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7ec9e1e6{/solr,null,AVAILABLE}
   [junit4]   2> 2862133 INFO  (jetty-launcher-5526-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2862133 INFO  (jetty-launcher-5526-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2862133 INFO  (jetty-launcher-5526-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2862134 INFO  (jetty-launcher-5526-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7eb03d29{/solr,null,AVAILABLE}
   [junit4]   2> 2862134 INFO  (jetty-launcher-5526-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@616caff6{HTTP/1.1,[http/1.1]}{127.0.0.1:53241}
   [junit4]   2> 2862134 INFO  (jetty-launcher-5526-thread-2) [    ] o.e.j.s.Server Started @2862254ms
   [junit4]   2> 2862134 INFO  (jetty-launcher-5526-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53241}
   [junit4]   2> 2862135 ERROR (jetty-launcher-5526-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2862135 INFO  (jetty-launcher-5526-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2862135 INFO  (jetty-launcher-5526-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.4.0
   [junit4]   2> 2862135 INFO  (jetty-launcher-5526-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2862135 INFO  (jetty-launcher-5526-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2862135 INFO  (jetty-launcher-5526-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-05-21T19:44:54.740Z
   [junit4]   2> 2862143 INFO  (jetty-launcher-5526-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2cfdbe7f{HTTP/1.1,[http/1.1]}{127.0.0.1:38720}
   [junit4]   2> 2862143 INFO  (jetty-launcher-5526-thread-1) [    ] o.e.j.s.Server Started @2862263ms
   [junit4]   2> 2862143 INFO  (jetty-launcher-5526-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38720}
   [junit4]   2> 2862144 ERROR (jetty-launcher-5526-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2862144 INFO  (jetty-launcher-5526-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2862144 INFO  (jetty-launcher-5526-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.4.0
   [junit4]   2> 2862144 INFO  (jetty-launcher-5526-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2862144 INFO  (jetty-launcher-5526-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2862144 INFO  (jetty-launcher-5526-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-05-21T19:44:54.749Z
   [junit4]   2> 2862182 INFO  (zkConnectionManagerCallback-5531-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2862182 INFO  (jetty-launcher-5526-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2862202 INFO  (zkConnectionManagerCallback-5533-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2862203 INFO  (jetty-launcher-5526-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2862227 INFO  (jetty-launcher-5526-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51983/solr
   [junit4]   2> 2862248 INFO  (jetty-launcher-5526-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51983/solr
   [junit4]   2> 2862256 INFO  (zkConnectionManagerCallback-5537-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2862258 INFO  (zkConnectionManagerCallback-5541-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2862301 INFO  (zkConnectionManagerCallback-5543-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2862336 INFO  (zkConnectionManagerCallback-5545-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2862348 INFO  (jetty-launcher-5526-thread-1) [n:127.0.0.1:38720_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2862351 INFO  (jetty-launcher-5526-thread-1) [n:127.0.0.1:38720_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38720_solr
   [junit4]   2> 2862352 INFO  (jetty-launcher-5526-thread-1) [n:127.0.0.1:38720_solr    ] o.a.s.c.Overseer Overseer (id=73839110996361222-127.0.0.1:38720_solr-n_0000000000) starting
   [junit4]   2> 2862409 INFO  (zkConnectionManagerCallback-5554-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2862412 INFO  (jetty-launcher-5526-thread-1) [n:127.0.0.1:38720_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:51983/solr ready
   [junit4]   2> 2862443 INFO  (jetty-launcher-5526-thread-2) [n:127.0.0.1:53241_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2862443 INFO  (jetty-launcher-5526-thread-1) [n:127.0.0.1:38720_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38720_solr
   [junit4]   2> 2862459 INFO  (zkCallback-5544-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2862460 INFO  (jetty-launcher-5526-thread-2) [n:127.0.0.1:53241_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2862460 INFO  (jetty-launcher-5526-thread-2) [n:127.0.0.1:53241_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53241_solr
   [junit4]   2> 2862479 INFO  (zkCallback-5553-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2862494 INFO  (zkCallback-5544-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2862507 INFO  (zkCallback-5542-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2863017 INFO  (jetty-launcher-5526-thread-1) [n:127.0.0.1:38720_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin
   [junit4]   2> 2863173 INFO  (jetty-launcher-5526-thread-1) [n:127.0.0.1:38720_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38720.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@19e39d98
   [junit4]   2> 2863232 INFO  (jetty-launcher-5526-thread-1) [n:127.0.0.1:38720_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38720.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@19e39d98
   [junit4]   2> 2863232 INFO  (jetty-launcher-5526-thread-1) [n:127.0.0.1:38720_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38720.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@19e39d98
   [junit4]   2> 2863237 INFO  (jetty-launcher-5526-thread-1) [n:127.0.0.1:38720_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_E67EF89C67B9ADC0-001/tempDir-001/node1/.
   [junit4]   2> 2863334 INFO  (jetty-launcher-5526-thread-2) [n:127.0.0.1:53241_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin
   [junit4]   2> 2863416 INFO  (jetty-launcher-5526-thread-2) [n:127.0.0.1:53241_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53241.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@19e39d98
   [junit4]   2> 2863441 INFO  (jetty-launcher-5526-thread-2) [n:127.0.0.1:53241_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53241.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@19e39d98
   [junit4]   2> 2863442 INFO  (jetty-launcher-5526-thread-2) [n:127.0.0.1:53241_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53241.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@19e39d98
   [junit4]   2> 2863463 INFO  (jetty-launcher-5526-thread-2) [n:127.0.0.1:53241_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_E67EF89C67B9ADC0-001/tempDir-001/node2/.
   [junit4]   2> 2863658 INFO  (zkConnectionManagerCallback-5559-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2863659 INFO  (jetty-launcher-5526-thread-2) [n:127.0.0.1:53241_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2863660 INFO  (jetty-launcher-5526-thread-2) [n:127.0.0.1:53241_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:51983/solr ready
   [junit4]   2> 2863718 INFO  (zkConnectionManagerCallback-5561-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2863759 INFO  (zkConnectionManagerCallback-5566-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2863760 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[E67EF89C67B9ADC0]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2863762 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[E67EF89C67B9ADC0]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:51983/solr ready
   [junit4]   2> 2863773 INFO  (qtp2107655772-21815) [n:127.0.0.1:38720_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=solrCloudCollectionConfig&maxShardsPerNode=2&name=testcollection&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2863790 INFO  (OverseerThreadFactory-7512-thread-1) [    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testcollection
   [junit4]   2> 2863900 INFO  (OverseerStateUpdate-73839110996361222-127.0.0.1:38720_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testcollection_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:53241/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2863903 INFO  (OverseerStateUpdate-73839110996361222-127.0.0.1:38720_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testcollection_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:38720/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2863906 INFO  (OverseerStateUpdate-73839110996361222-127.0.0.1:38720_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testcollection_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:53241/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2863916 INFO  (OverseerStateUpdate-73839110996361222-127.0.0.1:38720_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testcollection_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:38720/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2864199 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr    x:testcollection_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=solrCloudCollectionConfig&newCollection=true&name=testcollection_shard1_replica_n2&action=CREATE&numShards=2&collection=testcollection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2864200 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr    x:testcollection_shard1_replica_n2] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2864215 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr    x:testcollection_shard2_replica_n6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=solrCloudCollectionConfig&newCollection=true&name=testcollection_shard2_replica_n6&action=CREATE&numShards=2&collection=testcollection&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2864215 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr    x:testcollection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=solrCloudCollectionConfig&newCollection=true&name=testcollection_shard1_replica_n1&action=CREATE&numShards=2&collection=testcollection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2864238 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr    x:testcollection_shard2_replica_n4] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=solrCloudCollectionConfig&newCollection=true&name=testcollection_shard2_replica_n4&action=CREATE&numShards=2&collection=testcollection&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2865240 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2865250 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2865266 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2865283 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2865322 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.s.IndexSchema [testcollection_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 2865328 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.s.IndexSchema [testcollection_shard2_replica_n4] Schema name=minimal
   [junit4]   2> 2865333 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.s.IndexSchema [testcollection_shard2_replica_n6] Schema name=minimal
   [junit4]   2> 2865338 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.s.IndexSchema [testcollection_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 2865341 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2865341 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testcollection_shard1_replica_n2' using configuration from collection testcollection, trusted=true
   [junit4]   2> 2865341 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38720.solr.core.testcollection.shard1.replica_n2' (registry 'solr.core.testcollection.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@19e39d98
   [junit4]   2> 2865347 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2865347 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'testcollection_shard2_replica_n4' using configuration from collection testcollection, trusted=true
   [junit4]   2> 2865350 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2865350 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'testcollection_shard2_replica_n6' using configuration from collection testcollection, trusted=true
   [junit4]   2> 2865351 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2865351 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testcollection_shard1_replica_n1' using configuration from collection testcollection, trusted=true
   [junit4]   2> 2865351 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2865351 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.SolrCore [[testcollection_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_E67EF89C67B9ADC0-001/tempDir-001/node1/testcollection_shard1_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_E67EF89C67B9ADC0-001/tempDir-001/node1/./testcollection_shard1_replica_n2/data/]
   [junit4]   2> 2865354 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53241.solr.core.testcollection.shard1.replica_n1' (registry 'solr.core.testcollection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@19e39d98
   [junit4]   2> 2865354 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38720.solr.core.testcollection.shard2.replica_n6' (registry 'solr.core.testcollection.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@19e39d98
   [junit4]   2> 2865354 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2865355 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.SolrCore [[testcollection_shard2_replica_n6] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_E67EF89C67B9ADC0-001/tempDir-001/node1/testcollection_shard2_replica_n6], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_E67EF89C67B9ADC0-001/tempDir-001/node1/./testcollection_shard2_replica_n6/data/]
   [junit4]   2> 2865357 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53241.solr.core.testcollection.shard2.replica_n4' (registry 'solr.core.testcollection.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@19e39d98
   [junit4]   2> 2865357 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2865357 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.SolrCore [[testcollection_shard2_replica_n4] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_E67EF89C67B9ADC0-001/tempDir-001/node2/testcollection_shard2_replica_n4], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_E67EF89C67B9ADC0-001/tempDir-001/node2/./testcollection_shard2_replica_n4/data/]
   [junit4]   2> 2865359 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2865359 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.SolrCore [[testcollection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_E67EF89C67B9ADC0-001/tempDir-001/node2/testcollection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_E67EF89C67B9ADC0-001/tempDir-001/node2/./testcollection_shard1_replica_n1/data/]
   [junit4]   2> 2865707 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2865707 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2865709 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2865709 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2865711 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@18b96960[testcollection_shard1_replica_n2] main]
   [junit4]   2> 2865713 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
   [junit4]   2> 2865713 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 2865714 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2865714 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1601104142211743744
   [junit4]   2> 2865716 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2865716 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2865716 INFO  (searcherExecutor-7517-thread-1-processing-n:127.0.0.1:38720_solr x:testcollection_shard1_replica_n2 c:testcollection s:shard1 r:core_node5) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.SolrCore [testcollection_shard1_replica_n2] Registered new searcher Searcher@18b96960[testcollection_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2865717 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2865717 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2865723 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.s.SolrIndexSearcher Opening [Searcher@152b3d8d[testcollection_shard2_replica_n4] main]
   [junit4]   2> 2865725 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
   [junit4]   2> 2865726 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 2865727 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2865733 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2865733 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2865735 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2865735 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2865736 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1601104142234812416
   [junit4]   2> 2865737 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6cad3dc0[testcollection_shard1_replica_n1] main]
   [junit4]   2> 2865738 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2865738 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2865739 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2865739 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2865742 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@7408a973[testcollection_shard2_replica_n6] main]
   [junit4]   2> 2865743 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
   [junit4]   2> 2865743 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 2865744 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2865745 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1601104142244249600
   [junit4]   2> 2865747 INFO  (searcherExecutor-7520-thread-1-processing-n:127.0.0.1:53241_solr x:testcollection_shard1_replica_n1 c:testcollection s:shard1 r:core_node3) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.SolrCore [testcollection_shard1_replica_n1] Registered new searcher Searcher@6cad3dc0[testcollection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2865750 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testcollection/terms/shard2 to Terms{values={core_node7=0}, version=0}
   [junit4]   2> 2865754 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testcollection/terms/shard1 to Terms{values={core_node3=0}, version=0}
   [junit4]   2> 2865755 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testcollection/terms/shard1 to Terms{values={core_node3=0, core_node5=0}, version=1}
   [junit4]   2> 2865756 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
   [junit4]   2> 2865756 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 2865759 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 2865760 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2865761 INFO  (searcherExecutor-7518-thread-1-processing-n:127.0.0.1:53241_solr x:testcollection_shard2_replica_n4 c:testcollection s:shard2 r:core_node7) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.SolrCore [testcollection_shard2_replica_n4] Registered new searcher Searcher@152b3d8d[testcollection_shard2_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2865762 INFO  (searcherExecutor-7519-thread-1-processing-n:127.0.0.1:38720_solr x:testcollection_shard2_replica_n6 c:testcollection s:shard2 r:core_node8) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.SolrCore [testcollection_shard2_replica_n6] Registered new searcher Searcher@7408a973[testcollection_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2865762 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 2865763 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1601104142263123968
   [junit4]   2> 2865771 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testcollection/terms/shard2 to Terms{values={core_node7=0, core_node8=0}, version=1}
   [junit4]   2> 2866261 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2866261 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2866261 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:38720/solr/testcollection_shard1_replica_n2/
   [junit4]   2> 2866262 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=testcollection_shard1_replica_n2 url=http://127.0.0.1:38720/solr START replicas=[http://127.0.0.1:53241/solr/testcollection_shard1_replica_n1/] nUpdates=100
   [junit4]   2> 2866263 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2866263 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2866264 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:53241/solr/testcollection_shard2_replica_n4/
   [junit4]   2> 2866264 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.PeerSync PeerSync: core=testcollection_shard2_replica_n4 url=http://127.0.0.1:53241/solr START replicas=[http://127.0.0.1:38720/solr/testcollection_shard2_replica_n6/] nUpdates=100
   [junit4]   2> 2866309 INFO  (qtp1366550337-21820) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.S.Request [testcollection_shard1_replica_n1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=30
   [junit4]   2> 2866328 INFO  (qtp2107655772-21814) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.S.Request [testcollection_shard2_replica_n6]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=11
   [junit4]   2> 2866576 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=testcollection_shard1_replica_n2 url=http://127.0.0.1:38720/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 2866577 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 2866577 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 2866577 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2866581 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:38720/solr/testcollection_shard1_replica_n2/ shard1
   [junit4]   2> 2866591 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.PeerSync PeerSync: core=testcollection_shard2_replica_n4 url=http://127.0.0.1:53241/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 2866591 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 2866591 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 2866591 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2866657 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:53241/solr/testcollection_shard2_replica_n4/ shard2
   [junit4]   2> 2866759 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2866762 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=solrCloudCollectionConfig&newCollection=true&name=testcollection_shard2_replica_n4&action=CREATE&numShards=2&collection=testcollection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2524
   [junit4]   2> 2866768 INFO  (qtp1366550337-21812) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=solrCloudCollectionConfig&newCollection=true&name=testcollection_shard1_replica_n1&action=CREATE&numShards=2&collection=testcollection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2552
   [junit4]   2> 2866779 INFO  (qtp2107655772-21809) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=solrCloudCollectionConfig&newCollection=true&name=testcollection_shard2_replica_n6&action=CREATE&numShards=2&collection=testcollection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2564
   [junit4]   2> 2866790 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2866792 INFO  (qtp2107655772-21808) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=solrCloudCollectionConfig&newCollection=true&name=testcollection_shard1_replica_n2&action=CREATE&numShards=2&collection=testcollection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2592
   [junit4]   2> 2866798 INFO  (qtp2107655772-21815) [n:127.0.0.1:38720_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2866896 INFO  (zkCallback-5542-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testcollection/state.json] for collection [testcollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2866897 INFO  (zkCallback-5544-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testcollection/state.json] for collection [testcollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2867793 INFO  (OverseerCollectionConfigSetProcessor-73839110996361222-127.0.0.1:38720_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2867798 INFO  (qtp2107655772-21815) [n:127.0.0.1:38720_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&collection.configName=solrCloudCollectionConfig&maxShardsPerNode=2&name=testcollection&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=4024
   [junit4]   2> 2867798 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[E67EF89C67B9ADC0]) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: testcollection failOnTimeout:true timeout (sec):330
   [junit4]   1> -
   [junit4]   1> replica:core_node3 rstate:active live:true
   [junit4]   1> replica:core_node5 rstate:active live:true
   [junit4]   1> replica:core_node7 rstate:active live:true
   [junit4]   1> replica:core_node8 rstate:active live:true
   [junit4]   1> no one is recoverying
   [junit4]   2> 2867799 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[E67EF89C67B9ADC0]) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: testcollection
   [junit4]   2> 2867806 INFO  (qtp1366550337-21820) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.S.Request [testcollection_shard2_replica_n4]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=testcollection:5&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://127.0.0.1:53241/solr/testcollection_shard2_replica_n4/|http://127.0.0.1:38720/solr/testcollection_shard2_replica_n6/&rows=10&version=2&q=*:*&NOW=1526931900405&isShard=true&wt=javabin} hits=0 status=0 QTime=0
   [junit4]   2> 2867807 INFO  (qtp2107655772-21816) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.S.Request [testcollection_shard1_replica_n2]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=testcollection:5&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://127.0.0.1:38720/solr/testcollection_shard1_replica_n2/|http://127.0.0.1:53241/solr/testcollection_shard1_replica_n1/&rows=10&version=2&q=*:*&NOW=1526931900405&isShard=true&wt=javabin} hits=0 status=0 QTime=0
   [junit4]   2> 2867807 INFO  (qtp2107655772-21817) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.S.Request [testcollection_shard2_replica_n6]  webapp=/solr path=/select params={q=*:*&_stateVer_=testcollection:5&wt=javabin&version=2} hits=0 status=0 QTime=7
   [junit4]   2> 2867819 INFO  (qtp2107655772-21902) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testcollection/terms/shard1 to Terms{values={core_node3=1, core_node5=1}, version=2}
   [junit4]   2> 2867830 INFO  (qtp1366550337-21810) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testcollection_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38720/solr/testcollection_shard1_replica_n2/&wt=javabin&version=2}{add=[1 (1601104144413753344)]} 0 1
   [junit4]   2> 2867830 INFO  (qtp2107655772-21902) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testcollection_shard1_replica_n2]  webapp=/solr path=/update params={_stateVer_=testcollection:5&wt=javabin&version=2}{add=[1 (1601104144413753344)]} 0 16
   [junit4]   2> 2867832 INFO  (qtp1366550337-21820) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1601104144432627712,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2867832 INFO  (qtp1366550337-21820) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@155a77c7 commitCommandVersion:1601104144432627712
   [junit4]   2> 2867843 INFO  (qtp2107655772-21816) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1601104144444162048,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2867843 INFO  (qtp2107655772-21902) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1601104144444162048,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2867843 INFO  (qtp2107655772-21816) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@aae9afc commitCommandVersion:1601104144444162048
   [junit4]   2> 2867843 INFO  (qtp2107655772-21902) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2867846 INFO  (qtp2107655772-21902) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2867846 INFO  (qtp2107655772-21902) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory [testcollection_shard2_replica_n6]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:38720/solr/testcollection_shard1_replica_n2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 3
   [junit4]   2> 2867847 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1601104144448356352,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2867850 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2867851 INFO  (qtp1366550337-21820) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6b098661[testcollection_shard1_replica_n1] main]
   [junit4]   2> 2867852 INFO  (qtp1366550337-21820) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2867853 INFO  (searcherExecutor-7520-thread-1-processing-n:127.0.0.1:53241_solr x:testcollection_shard1_replica_n1 c:testcollection s:shard1 r:core_node3) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.SolrCore [testcollection_shard1_replica_n1] Registered new searcher Searcher@6b098661[testcollection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.4.0):C1)))}
   [junit4]   2> 2867854 INFO  (qtp1366550337-21820) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testcollection_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:38720/solr/testcollection_shard1_replica_n2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 21
   [junit4]   2> 2867856 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2867859 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.p.LogUpdateProcessorFactory [testcollection_shard2_replica_n4]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:38720/solr/testcollection_shard1_replica_n2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 11
   [junit4]   2> 2867860 INFO  (qtp2107655772-21816) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@3d29f481[testcollection_shard1_replica_n2] main]
   [junit4]   2> 2867860 INFO  (qtp2107655772-21816) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2867862 INFO  (searcherExecutor-7517-thread-1-processing-n:127.0.0.1:38720_solr x:testcollection_shard1_replica_n2 c:testcollection s:shard1 r:core_node5) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.SolrCore [testcollection_shard1_replica_n2] Registered new searcher Searcher@3d29f481[testcollection_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.4.0):C1)))}
   [junit4]   2> 2867862 INFO  (qtp2107655772-21816) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testcollection_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:38720/solr/testcollection_shard1_replica_n2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 18
   [junit4]   2> 2867862 INFO  (qtp2107655772-21817) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testcollection_shard1_replica_n2]  webapp=/solr path=/update params={_stateVer_=testcollection:5&commit=true&wt=javabin&version=2}{commit=} 0 31
   [junit4]   2> 2867865 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.S.Request [testcollection_shard2_replica_n4]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=testcollection:5&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://127.0.0.1:53241/solr/testcollection_shard2_replica_n4/|http://127.0.0.1:38720/solr/testcollection_shard2_replica_n6/&rows=10&version=2&q=*:*&NOW=1526931900468&isShard=true&wt=javabin} hits=0 status=0 QTime=0
   [junit4]   2> 2867879 INFO  (qtp2107655772-21814) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.S.Request [testcollection_shard1_replica_n2]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=testcollection:5&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=http://127.0.0.1:38720/solr/testcollection_shard1_replica_n2/|http://127.0.0.1:53241/solr/testcollection_shard1_replica_n1/&rows=10&version=2&q=*:*&NOW=1526931900468&isShard=true&wt=javabin} hits=1 status=0 QTime=0
   [junit4]   2> 2867885 INFO  (qtp2107655772-21814) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.S.Request [testcollection_shard1_replica_n2]  webapp=/solr path=/select params={q=*:*&df=text&distrib=false&_stateVer_=testcollection:5&shards.purpose=64&NOW=1526931900468&ids=1&isShard=true&shard.url=http://127.0.0.1:38720/solr/testcollection_shard1_replica_n2/|http://127.0.0.1:53241/solr/testcollection_shard1_replica_n1/&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2867885 INFO  (qtp1366550337-21824) [n:127.0.0.1:53241_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.S.Request [testcollection_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&_stateVer_=testcollection:5&wt=javabin&version=2} hits=1 status=0 QTime=21
   [junit4]   2> 2867886 INFO  (qtp2107655772-21814) [n:127.0.0.1:38720_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=testcollection&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2867893 INFO  (OverseerThreadFactory-7512-thread-2) [    ] o.a.s.c.a.c.OverseerCollectionMessageHandler Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null
   [junit4]   2> 2867894 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr    x:testcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testcollection.shard1.replica_n1, tag=null
   [junit4]   2> 2867894 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr    x:testcollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5583a6a: rootName = solr_53241, domain = solr.core.testcollection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.testcollection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@6cb1cc1
   [junit4]   2> 2867894 INFO  (qtp2107655772-21903) [n:127.0.0.1:38720_solr    x:testcollection_shard2_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testcollection.shard2.replica_n6, tag=null
   [junit4]   2> 2867894 INFO  (qtp2107655772-21903) [n:127.0.0.1:38720_solr    x:testcollection_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@388b81e1: rootName = solr_38720, domain = solr.core.testcollection.shard2.replica_n6, service url = null, agent id = null] for registry solr.core.testcollection.shard2.replica_n6 / com.codahale.metrics.MetricRegistry@3d50e8f1
   [junit4]   2> 2867932 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr    x:testcollection_shard2_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testcollection.shard2.replica_n4, tag=null
   [junit4]   2> 2867932 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr    x:testcollection_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@717d7f0f: rootName = solr_53241, domain = solr.core.testcollection.shard2.replica_n4, service url = null, agent id = null] for registry solr.core.testcollection.shard2.replica_n4 / com.codahale.metrics.MetricRegistry@cbc204b
   [junit4]   2> 2867944 INFO  (qtp2107655772-21817) [n:127.0.0.1:38720_solr    x:testcollection_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testcollection.shard1.replica_n2, tag=null
   [junit4]   2> 2867944 INFO  (qtp2107655772-21817) [n:127.0.0.1:38720_solr    x:testcollection_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2b442f02: rootName = solr_38720, domain = solr.core.testcollection.shard1.replica_n2, service url = null, agent id = null] for registry solr.core.testcollection.shard1.replica_n2 / com.codahale.metrics.MetricRegistry@5c51f3ca
   [junit4]   2> 2867945 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr    x:testcollection_shard1_replica_n1] o.a.s.c.SolrCore [testcollection_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@26183978
   [junit4]   2> 2867954 INFO  (qtp2107655772-21903) [n:127.0.0.1:38720_solr    x:testcollection_shard2_replica_n6] o.a.s.c.SolrCore [testcollection_shard2_replica_n6]  CLOSING SolrCore org.apache.solr.core.SolrCore@3b59035c
   [junit4]   2> 2867970 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr    x:testcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testcollection.shard1.replica_n1, tag=26183978
   [junit4]   2> 2867970 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr    x:testcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testcollection.shard1.leader, tag=26183978
   [junit4]   2> 2867979 INFO  (qtp2107655772-21817) [n:127.0.0.1:38720_solr    x:testcollection_shard1_replica_n2] o.a.s.c.SolrCore [testcollection_shard1_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@4653f830
   [junit4]   2> 2867979 INFO  (qtp2107655772-21817) [n:127.0.0.1:38720_solr    x:testcollection_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testcollection.shard1.replica_n2, tag=4653f830
   [junit4]   2> 2867979 INFO  (qtp2107655772-21817) [n:127.0.0.1:38720_solr    x:testcollection_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testcollection.shard1.leader, tag=4653f830
   [junit4]   2> 2867979 INFO  (qtp2107655772-21903) [n:127.0.0.1:38720_solr    x:testcollection_shard2_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testcollection.shard2.replica_n6, tag=3b59035c
   [junit4]   2> 2867979 INFO  (qtp2107655772-21903) [n:127.0.0.1:38720_solr    x:testcollection_shard2_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testcollection.shard2.leader, tag=3b59035c
   [junit4]   2> 2867981 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr    x:testcollection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testcollection/terms/shard1 to Terms{values={core_node5=1}, version=3}
   [junit4]   2> 2867982 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr    x:testcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
   [junit4]   2> 2867983 INFO  (qtp1366550337-21823) [n:127.0.0.1:53241_solr    x:testcollection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=testcollection_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=89
   [junit4]   2> 2867986 INFO  (qtp2107655772-21903) [n:127.0.0.1:38720_solr    x:testcollection_shard2_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testcollection/terms/shard2 to Terms{values={core_node7=0}, version=2}
   [junit4]   2> 2867987 INFO  (qtp2107655772-21903) [n:127.0.0.1:38720_solr    x:testcollection_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
   [junit4]   2> 2867987 INFO  (qtp2107655772-21903) [n:127.0.0.1:38720_solr    x:testcollection_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=testcollection_shard2_replica_n6&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=93
   [junit4]   2> 2867988 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr    x:testcollection_shard2_replica_n4] o.a.s.c.SolrCore [testcollection_shard2_replica_n4]  CLOSING SolrCore org.apache.solr.core.SolrCore@575b194
   [junit4]   2> 2867988 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr    x:testcollection_shard2_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testcollection.shard2.replica_n4, tag=575b194
   [junit4]   2> 2867988 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr    x:testcollection_shard2_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testcollection.shard2.leader, tag=575b194
   [junit4]   2> 2868001 INFO  (qtp2107655772-21817) [n:127.0.0.1:38720_solr    x:testcollection_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testcollection/terms/shard1 to Terms{values={}, version=4}
   [junit4]   2> 2868002 WARN  (zkCallback-5542-thread-2) [    ] o.a.s.c.LeaderElector Our node is no longer in line to be leader
   [junit4]   2> 2868003 INFO  (qtp2107655772-21817) [n:127.0.0.1:38720_solr    x:testcollection_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=testcollection_shard1_replica_n2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=104
   [junit4]   2> 2868006 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr    x:testcollection_shard2_replica_n4] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testcollection/terms/shard2 to Terms{values={}, version=3}
   [junit4]   2> 2868007 WARN  (zkCallback-5544-thread-1) [    ] o.a.s.c.LeaderElector Our node is no longer in line to be leader
   [junit4]   2> 2868017 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr    x:testcollection_shard2_replica_n4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=testcollection_shard2_replica_n4&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=119
   [junit4]   2> 2868120 INFO  (zkCallback-5544-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/testcollection/state.json] for collection [testcollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2868121 INFO  (zkCallback-5542-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/testcollection/state.json] for collection [testcollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2868729 INFO  (qtp2107655772-21814) [n:127.0.0.1:38720_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=testcollection&action=DELETE&wt=javabin&version=2} status=0 QTime=843
   [junit4]   2> 2868729 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[E67EF89C67B9ADC0]) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for collection to disappear - collection: testcollection failOnTimeout:true timeout (sec):330
   [junit4]   1> -
   [junit4]   2> 2868729 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[E67EF89C67B9ADC0]) [    ] o.a.s.c.AbstractDistribZkTestBase Collection has disappeared - collection: testcollection
   [junit4]   2> 2868730 INFO  (qtp2107655772-21902) [n:127.0.0.1:38720_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params async=e97bc853-f476-43d0-8722-249ffd95baa4&replicationFactor=2&collection.configName=solrCloudCollectionConfig&maxShardsPerNode=2&name=testcollection&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2868732 INFO  (qtp2107655772-21902) [n:127.0.0.1:38720_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={async=e97bc853-f476-43d0-8722-249ffd95baa4&replicationFactor=2&collection.configName=solrCloudCollectionConfig&maxShardsPerNode=2&name=testcollection&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 2868732 INFO  (qtp2107655772-21814) [n:127.0.0.1:38720_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=e97bc853-f476-43d0-8722-249ffd95baa4&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2868734 INFO  (qtp2107655772-21814) [n:127.0.0.1:38720_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=e97bc853-f476-43d0-8722-249ffd95baa4&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 2868750 INFO  (OverseerThreadFactory-7512-thread-3) [    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testcollection
   [junit4]   2> 2868751 INFO  (OverseerCollectionConfigSetProcessor-73839110996361222-127.0.0.1:38720_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2868969 INFO  (OverseerStateUpdate-73839110996361222-127.0.0.1:38720_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testcollection_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:53241/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2868973 INFO  (OverseerStateUpdate-73839110996361222-127.0.0.1:38720_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testcollection_shard1_replica_n3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:38720/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2868976 INFO  (OverseerStateUpdate-73839110996361222-127.0.0.1:38720_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testcollection_shard2_replica_n5",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:53241/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2868991 INFO  (OverseerStateUpdate-73839110996361222-127.0.0.1:38720_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testcollection_shard2_replica_n7",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:38720/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2869306 INFO  (qtp2107655772-21902) [n:127.0.0.1:38720_solr    x:testcollection_shard1_replica_n3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=e97bc853-f476-43d0-8722-249ffd95baa427183798618683235&coreNodeName=core_node4&name=testcollection_shard1_replica_n3&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=0 QTime=11
   [junit4]   2> 2869306 INFO  (parallelCoreAdminExecutor-7513-thread-1-processing-n:127.0.0.1:38720_solr x:testcollection_shard1_replica_n3 e97bc853-f476-43d0-8722-249ffd95baa427183798618683235 CREATE) [n:127.0.0.1:38720_solr    x:testcollection_shard1_replica_n3] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=e97bc853-f476-43d0-8722-249ffd95baa427183798618683235&coreNodeName=core_node4&name=testcollection_shard1_replica_n3&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 2869307 INFO  (qtp1366550337-21824) [n:127.0.0.1:53241_solr    x:testcollection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=e97bc853-f476-43d0-8722-249ffd95baa427183798615071252&coreNodeName=core_node2&name=testcollection_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=0 QTime=11
   [junit4]   2> 2869307 INFO  (parallelCoreAdminExecutor-7515-thread-2-processing-n:127.0.0.1:53241_solr x:testcollection_shard1_replica_n1 e97bc853-f476-43d0-8722-249ffd95baa427183798615071252 CREATE) [n:127.0.0.1:53241_solr    x:testcollection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=e97bc853-f476-43d0-8722-249ffd95baa427183798615071252&coreNodeName=core_node2&name=testcollection_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 2869311 INFO  (qtp2107655772-21816) [n:127.0.0.1:38720_solr    x:testcollection_shard2_replica_n7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=e97bc853-f476-43d0-8722-249ffd95baa427183798636948892&coreNodeName=core_node8&name=testcollection_shard2_replica_n7&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=0 QTime=16
   [junit4]   2> 2869311 INFO  (parallelCoreAdminExecutor-7513-thread-2-processing-n:127.0.0.1:38720_solr x:testcollection_shard2_replica_n7 e97bc853-f476-43d0-8722-249ffd95baa427183798636948892 CREATE) [n:127.0.0.1:38720_solr    x:testcollection_shard2_replica_n7] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=e97bc853-f476-43d0-8722-249ffd95baa427183798636948892&coreNodeName=core_node8&name=testcollection_shard2_replica_n7&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 2869327 INFO  (qtp1366550337-21820) [n:127.0.0.1:53241_solr    x:testcollection_shard2_replica_n5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=e97bc853-f476-43d0-8722-249ffd95baa427183798622141627&coreNodeName=core_node6&name=testcollection_shard2_replica_n5&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=0 QTime=32
   [junit4]   2> 2869327 INFO  (parallelCoreAdminExecutor-7515-thread-1-processing-n:127.0.0.1:53241_solr x:testcollection_shard2_replica_n5 e97bc853-f476-43d0-8722-249ffd95baa427183798622141627 CREATE) [n:127.0.0.1:53241_solr    x:testcollection_shard2_replica_n5] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=e97bc853-f476-43d0-8722-249ffd95baa427183798622141627&coreNodeName=core_node6&name=testcollection_shard2_replica_n5&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 2869328 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : e97bc853-f476-43d0-8722-249ffd95baa427183798622141627
   [junit4]   2> 2869328 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=e97bc853-f476-43d0-8722-249ffd95baa427183798622141627&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2869735 INFO  (qtp2107655772-21814) [n:127.0.0.1:38720_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=e97bc853-f476-43d0-8722-249ffd95baa4&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2869736 INFO  (qtp2107655772-21814) [n:127.0.0.1:38720_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=e97bc853-f476-43d0-8722-249ffd95baa4&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 2870321 INFO  (parallelCoreAdminExecutor-7513-thread-1-processing-n:127.0.0.1:38720_solr x:testcollection_shard1_replica_n3 e97bc853-f476-43d0-8722-249ffd95baa427183798618683235 CREATE) [n:127.0.0.1:38720_solr c:testcollection s:shard1 r:core_node4 x:testcollection_shard1_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2870329 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : e97bc853-f476-43d0-8722-249ffd95baa427183798622141627
   [junit4]   2> 2870330 INFO  (qtp1366550337-21822) [n:127.0.0.1:53241_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=e97bc853-f476-43d0-8722-249ffd95baa427183798622141627&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2870342 INFO  (parallelCoreAdminExecutor-7513-thread-2-processing-n:127.0.0.1:38720_solr x:testcollection_shard2_replica_n7 e97bc853-f476-43d0-8722-249ffd95baa427183798636948892 CREATE) [n:127.0.0.1:38720_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n7] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2870388 INFO  (parallelCoreAdminExecutor-7515-thread-1-processing-n:127.0.0.1:53241_solr x:testcollection_shard2_replica_n5 e97bc853-f476-43d0-8722-249ffd95baa427183798622141627 CREATE) [n:127.0.0.1:53241_solr c:testcollection s:shard2 r:core_node6 x:testcollection_shard2_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2870393 INFO  (parallelCoreAdminExecutor-7515-thread-2-processing-n:127.0.0.1:53241_solr x:testcollection_shard1_replica_n1 e97bc853-f476-43d0-8722-249ffd95baa427183

[...truncated too long message...]

owed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null323693840
     [copy] Copying 39 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null323693840
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null323693840

resolve-example:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve-server:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1948597150
     [copy] Copying 246 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1948597150
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1948597150

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] 	confs: [default]
[ivy:cachepath] 	found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] 	found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] 	found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] 	found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] 	found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] 	found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] 	found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] 	found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 62ms :: artifacts dl 6ms
	---------------------------------------------------------------------
	|                  |            modules            ||   artifacts   |
	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
	---------------------------------------------------------------------
	|      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
	---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 270 minutes 52 seconds
Archiving artifacts
WARN: No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
WARN: java.lang.InterruptedException: no matches found within 10000
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)

[JENKINS] Lucene-Solr-BadApples-Tests-7.x - Build # 67 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/67/

1 tests failed.
FAILED:  org.apache.solr.cloud.TestAuthenticationFramework.testBasics

Error Message:
Error from server at https://127.0.0.1:37910/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason: <pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.10.v20180503</a><hr/>  </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:37910/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.10.v20180503</a><hr/>

</body>
</html>

	at __randomizedtesting.SeedInfo.seed([970938951E7E6CC1:AAD196B9269032B1]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1015)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
	at org.apache.solr.cloud.TestAuthenticationFramework.collectionCreateSearchDeleteTwice(TestAuthenticationFramework.java:127)
	at org.apache.solr.cloud.TestAuthenticationFramework.testBasics(TestAuthenticationFramework.java:75)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:37910/solr/testcollection_shard1_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n2/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.10.v20180503</a><hr/>

</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:202)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more




Build Log:
[...truncated 14236 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestAuthenticationFramework
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_970938951E7E6CC1-001/init-core-data-001
   [junit4]   2> 2964077 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[970938951E7E6CC1]) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_970938951E7E6CC1-001/tempDir-001
   [junit4]   2> 2964078 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[970938951E7E6CC1]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2964078 INFO  (Thread-5301) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2964078 INFO  (Thread-5301) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2964083 ERROR (Thread-5301) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2964178 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[970938951E7E6CC1]) [    ] o.a.s.c.ZkTestServer start zk server on port:59101
   [junit4]   2> 2964198 INFO  (zkConnectionManagerCallback-5200-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2964213 INFO  (jetty-launcher-5197-thread-1) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
   [junit4]   2> 2964221 INFO  (jetty-launcher-5197-thread-2) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
   [junit4]   2> 2964226 INFO  (jetty-launcher-5197-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2964226 INFO  (jetty-launcher-5197-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2964226 INFO  (jetty-launcher-5197-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2964227 INFO  (jetty-launcher-5197-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7d36a0d3{/solr,null,AVAILABLE}
   [junit4]   2> 2964227 INFO  (jetty-launcher-5197-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4ba6d803{SSL,[ssl, http/1.1]}{127.0.0.1:41228}
   [junit4]   2> 2964227 INFO  (jetty-launcher-5197-thread-2) [    ] o.e.j.s.Server Started @2964283ms
   [junit4]   2> 2964227 INFO  (jetty-launcher-5197-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41228}
   [junit4]   2> 2964227 ERROR (jetty-launcher-5197-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2964228 INFO  (jetty-launcher-5197-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2964228 INFO  (jetty-launcher-5197-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.4.0
   [junit4]   2> 2964228 INFO  (jetty-launcher-5197-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2964228 INFO  (jetty-launcher-5197-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2964228 INFO  (jetty-launcher-5197-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-05-26T04:39:10.433Z
   [junit4]   2> 2964233 INFO  (jetty-launcher-5197-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2964233 INFO  (jetty-launcher-5197-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2964233 INFO  (jetty-launcher-5197-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2964233 INFO  (jetty-launcher-5197-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@49d23400{/solr,null,AVAILABLE}
   [junit4]   2> 2964234 INFO  (jetty-launcher-5197-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4dcd7982{SSL,[ssl, http/1.1]}{127.0.0.1:37910}
   [junit4]   2> 2964234 INFO  (jetty-launcher-5197-thread-1) [    ] o.e.j.s.Server Started @2964290ms
   [junit4]   2> 2964234 INFO  (jetty-launcher-5197-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37910}
   [junit4]   2> 2964234 ERROR (jetty-launcher-5197-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2964234 INFO  (jetty-launcher-5197-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2964234 INFO  (jetty-launcher-5197-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.4.0
   [junit4]   2> 2964234 INFO  (jetty-launcher-5197-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2964234 INFO  (jetty-launcher-5197-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2964234 INFO  (jetty-launcher-5197-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-05-26T04:39:10.439Z
   [junit4]   2> 2964237 INFO  (zkConnectionManagerCallback-5204-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2964237 INFO  (jetty-launcher-5197-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2964245 INFO  (zkConnectionManagerCallback-5202-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2964246 INFO  (jetty-launcher-5197-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2964255 INFO  (jetty-launcher-5197-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59101/solr
   [junit4]   2> 2964257 INFO  (jetty-launcher-5197-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59101/solr
   [junit4]   2> 2964260 INFO  (zkConnectionManagerCallback-5212-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2964261 INFO  (zkConnectionManagerCallback-5208-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2964264 INFO  (zkConnectionManagerCallback-5214-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2964278 INFO  (zkConnectionManagerCallback-5216-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2964300 INFO  (jetty-launcher-5197-thread-2) [n:127.0.0.1:41228_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2964301 INFO  (jetty-launcher-5197-thread-2) [n:127.0.0.1:41228_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41228_solr
   [junit4]   2> 2964302 INFO  (jetty-launcher-5197-thread-2) [n:127.0.0.1:41228_solr    ] o.a.s.c.Overseer Overseer (id=73863861049819141-127.0.0.1:41228_solr-n_0000000000) starting
   [junit4]   2> 2964342 INFO  (zkConnectionManagerCallback-5225-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2964345 INFO  (jetty-launcher-5197-thread-2) [n:127.0.0.1:41228_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59101/solr ready
   [junit4]   2> 2964369 INFO  (jetty-launcher-5197-thread-1) [n:127.0.0.1:37910_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2964373 INFO  (jetty-launcher-5197-thread-1) [n:127.0.0.1:37910_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37910_solr
   [junit4]   2> 2964375 INFO  (jetty-launcher-5197-thread-2) [n:127.0.0.1:41228_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2964375 INFO  (jetty-launcher-5197-thread-2) [n:127.0.0.1:41228_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41228_solr
   [junit4]   2> 2964400 INFO  (zkCallback-5213-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2964410 INFO  (zkCallback-5224-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2964427 INFO  (zkCallback-5215-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2964552 INFO  (jetty-launcher-5197-thread-1) [n:127.0.0.1:37910_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin
   [junit4]   2> 2964614 INFO  (jetty-launcher-5197-thread-1) [n:127.0.0.1:37910_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37910.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3af45ffb
   [junit4]   2> 2964625 INFO  (jetty-launcher-5197-thread-1) [n:127.0.0.1:37910_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37910.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3af45ffb
   [junit4]   2> 2964626 INFO  (jetty-launcher-5197-thread-1) [n:127.0.0.1:37910_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37910.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3af45ffb
   [junit4]   2> 2964627 INFO  (jetty-launcher-5197-thread-1) [n:127.0.0.1:37910_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_970938951E7E6CC1-001/tempDir-001/node1/.
   [junit4]   2> 2964631 INFO  (zkConnectionManagerCallback-5230-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2964632 INFO  (jetty-launcher-5197-thread-1) [n:127.0.0.1:37910_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2964633 INFO  (jetty-launcher-5197-thread-1) [n:127.0.0.1:37910_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59101/solr ready
   [junit4]   2> 2964809 INFO  (jetty-launcher-5197-thread-2) [n:127.0.0.1:41228_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin
   [junit4]   2> 2964848 INFO  (jetty-launcher-5197-thread-2) [n:127.0.0.1:41228_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41228.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3af45ffb
   [junit4]   2> 2964860 INFO  (jetty-launcher-5197-thread-2) [n:127.0.0.1:41228_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41228.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3af45ffb
   [junit4]   2> 2964860 INFO  (jetty-launcher-5197-thread-2) [n:127.0.0.1:41228_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41228.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3af45ffb
   [junit4]   2> 2964861 INFO  (jetty-launcher-5197-thread-2) [n:127.0.0.1:41228_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_970938951E7E6CC1-001/tempDir-001/node2/.
   [junit4]   2> 2964893 INFO  (zkConnectionManagerCallback-5232-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2964897 INFO  (zkConnectionManagerCallback-5237-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2964898 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[970938951E7E6CC1]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2964899 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[970938951E7E6CC1]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59101/solr ready
   [junit4]   2> 2964933 INFO  (qtp197383304-22135) [n:127.0.0.1:37910_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params async=0ae7e8fb-c927-4cd6-b12a-b53dd2655045&replicationFactor=2&collection.configName=solrCloudCollectionConfig&maxShardsPerNode=2&name=testcollection&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2964935 INFO  (qtp197383304-22135) [n:127.0.0.1:37910_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={async=0ae7e8fb-c927-4cd6-b12a-b53dd2655045&replicationFactor=2&collection.configName=solrCloudCollectionConfig&maxShardsPerNode=2&name=testcollection&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 2964939 INFO  (qtp197383304-22137) [n:127.0.0.1:37910_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=0ae7e8fb-c927-4cd6-b12a-b53dd2655045&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2964940 INFO  (qtp197383304-22137) [n:127.0.0.1:37910_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=0ae7e8fb-c927-4cd6-b12a-b53dd2655045&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2964941 INFO  (OverseerThreadFactory-8064-thread-1) [    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testcollection
   [junit4]   2> 2965049 INFO  (OverseerStateUpdate-73863861049819141-127.0.0.1:41228_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testcollection_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:41228/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2965052 INFO  (OverseerStateUpdate-73863861049819141-127.0.0.1:41228_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testcollection_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:37910/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2965055 INFO  (OverseerStateUpdate-73863861049819141-127.0.0.1:41228_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testcollection_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:41228/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2965058 INFO  (OverseerStateUpdate-73863861049819141-127.0.0.1:41228_solr-n_0000000000) [    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testcollection",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testcollection_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:37910/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2965272 INFO  (qtp197383304-22137) [n:127.0.0.1:37910_solr    x:testcollection_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048&coreNodeName=core_node8&name=testcollection_shard2_replica_n6&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=0 QTime=0
   [junit4]   2> 2965273 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr    x:testcollection_shard2_replica_n6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048&coreNodeName=core_node8&name=testcollection_shard2_replica_n6&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 2965273 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr    x:testcollection_shard2_replica_n6] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2965276 INFO  (qtp197383304-22138) [n:127.0.0.1:37910_solr    x:testcollection_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003&coreNodeName=core_node5&name=testcollection_shard1_replica_n2&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=0 QTime=4
   [junit4]   2> 2965281 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr    x:testcollection_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003&coreNodeName=core_node5&name=testcollection_shard1_replica_n2&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 2965324 INFO  (qtp1032987856-22125) [n:127.0.0.1:41228_solr    x:testcollection_shard2_replica_n4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187&coreNodeName=core_node7&name=testcollection_shard2_replica_n4&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=0 QTime=0
   [junit4]   2> 2965325 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr    x:testcollection_shard2_replica_n4] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187&coreNodeName=core_node7&name=testcollection_shard2_replica_n4&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 2965327 INFO  (qtp1032987856-22129) [n:127.0.0.1:41228_solr    x:testcollection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833&coreNodeName=core_node3&name=testcollection_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=0 QTime=1
   [junit4]   2> 2965328 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr    x:testcollection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=testcollection&version=2&replicaType=NRT&async=0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833&coreNodeName=core_node3&name=testcollection_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 2965337 INFO  (qtp197383304-22132) [n:127.0.0.1:37910_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048
   [junit4]   2> 2965337 INFO  (qtp197383304-22132) [n:127.0.0.1:37910_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2965941 INFO  (qtp197383304-22135) [n:127.0.0.1:37910_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=0ae7e8fb-c927-4cd6-b12a-b53dd2655045&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2965942 INFO  (qtp197383304-22135) [n:127.0.0.1:37910_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=0ae7e8fb-c927-4cd6-b12a-b53dd2655045&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 2966309 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2966309 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2966317 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.s.IndexSchema [testcollection_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 2966317 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.s.IndexSchema [testcollection_shard2_replica_n6] Schema name=minimal
   [junit4]   2> 2966321 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2966321 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2966321 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'testcollection_shard2_replica_n6' using configuration from collection testcollection, trusted=true
   [junit4]   2> 2966321 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testcollection_shard1_replica_n2' using configuration from collection testcollection, trusted=true
   [junit4]   2> 2966321 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37910.solr.core.testcollection.shard1.replica_n2' (registry 'solr.core.testcollection.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3af45ffb
   [junit4]   2> 2966321 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37910.solr.core.testcollection.shard2.replica_n6' (registry 'solr.core.testcollection.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3af45ffb
   [junit4]   2> 2966321 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2966321 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2966321 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.SolrCore [[testcollection_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_970938951E7E6CC1-001/tempDir-001/node1/testcollection_shard1_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_970938951E7E6CC1-001/tempDir-001/node1/./testcollection_shard1_replica_n2/data/]
   [junit4]   2> 2966321 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.SolrCore [[testcollection_shard2_replica_n6] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_970938951E7E6CC1-001/tempDir-001/node1/testcollection_shard2_replica_n6], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_970938951E7E6CC1-001/tempDir-001/node1/./testcollection_shard2_replica_n6/data/]
   [junit4]   2> 2966343 INFO  (qtp197383304-22138) [n:127.0.0.1:37910_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048
   [junit4]   2> 2966343 INFO  (qtp197383304-22138) [n:127.0.0.1:37910_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2966347 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2966380 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 2966396 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.s.IndexSchema [testcollection_shard2_replica_n4] Schema name=minimal
   [junit4]   2> 2966399 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2966399 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'testcollection_shard2_replica_n4' using configuration from collection testcollection, trusted=true
   [junit4]   2> 2966400 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.s.IndexSchema [testcollection_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 2966407 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41228.solr.core.testcollection.shard2.replica_n4' (registry 'solr.core.testcollection.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3af45ffb
   [junit4]   2> 2966407 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2966407 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.SolrCore [[testcollection_shard2_replica_n4] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_970938951E7E6CC1-001/tempDir-001/node2/testcollection_shard2_replica_n4], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_970938951E7E6CC1-001/tempDir-001/node2/./testcollection_shard2_replica_n4/data/]
   [junit4]   2> 2966411 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2966411 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testcollection_shard1_replica_n1' using configuration from collection testcollection, trusted=true
   [junit4]   2> 2966411 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41228.solr.core.testcollection.shard1.replica_n1' (registry 'solr.core.testcollection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3af45ffb
   [junit4]   2> 2966411 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2966411 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.SolrCore [[testcollection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_970938951E7E6CC1-001/tempDir-001/node2/testcollection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestAuthenticationFramework_970938951E7E6CC1-001/tempDir-001/node2/./testcollection_shard1_replica_n1/data/]
   [junit4]   2> 2966464 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2966464 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2966465 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2966465 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2966474 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@18d23f45[testcollection_shard1_replica_n2] main]
   [junit4]   2> 2966475 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
   [junit4]   2> 2966476 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 2966476 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2966477 INFO  (searcherExecutor-8070-thread-1-processing-x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 r:core_node5 CREATE n:127.0.0.1:37910_solr c:testcollection s:shard1) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.SolrCore [testcollection_shard1_replica_n2] Registered new searcher Searcher@18d23f45[testcollection_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2966478 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1601500141514129408
   [junit4]   2> 2966481 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2966481 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2966483 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2966483 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2966484 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@2301ade3[testcollection_shard2_replica_n6] main]
   [junit4]   2> 2966486 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
   [junit4]   2> 2966486 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testcollection/terms/shard1 to Terms{values={core_node5=0}, version=0}
   [junit4]   2> 2966486 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 2966487 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2966487 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1601500141523566592
   [junit4]   2> 2966489 INFO  (searcherExecutor-8069-thread-1-processing-x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 r:core_node8 CREATE n:127.0.0.1:37910_solr c:testcollection s:shard2) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.SolrCore [testcollection_shard2_replica_n6] Registered new searcher Searcher@2301ade3[testcollection_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2966491 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 2966493 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testcollection/terms/shard2 to Terms{values={core_node8=0}, version=0}
   [junit4]   2> 2966500 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 2966543 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2966543 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2966543 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2966543 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2966544 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2966544 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2966544 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2966544 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2966546 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.s.SolrIndexSearcher Opening [Searcher@2a25033f[testcollection_shard2_replica_n4] main]
   [junit4]   2> 2966546 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@15076a25[testcollection_shard1_replica_n1] main]
   [junit4]   2> 2966547 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
   [junit4]   2> 2966548 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 2966549 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2966549 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1601500141588578304
   [junit4]   2> 2966550 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
   [junit4]   2> 2966551 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 2966552 INFO  (searcherExecutor-8072-thread-1-processing-x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 r:core_node3 CREATE n:127.0.0.1:41228_solr c:testcollection s:shard1) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.SolrCore [testcollection_shard1_replica_n1] Registered new searcher Searcher@15076a25[testcollection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2966554 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2966555 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1601500141594869760
   [junit4]   2> 2966556 INFO  (searcherExecutor-8071-thread-1-processing-x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 r:core_node7 CREATE n:127.0.0.1:41228_solr c:testcollection s:shard2) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.SolrCore [testcollection_shard2_replica_n4] Registered new searcher Searcher@2a25033f[testcollection_shard2_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2966559 INFO  (parallelCoreAdminExecutor-8067-thread-2-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448294744833 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testcollection/terms/shard1 to Terms{values={core_node3=0, core_node5=0}, version=1}
   [junit4]   2> 2966563 INFO  (parallelCoreAdminExecutor-8067-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard2_replica_n4 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187 CREATE) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testcollection/terms/shard2 to Terms{values={core_node7=0, core_node8=0}, version=1}
   [junit4]   2> 2966943 INFO  (qtp197383304-22132) [n:127.0.0.1:37910_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=0ae7e8fb-c927-4cd6-b12a-b53dd2655045&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2966944 INFO  (qtp197383304-22132) [n:127.0.0.1:37910_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=0ae7e8fb-c927-4cd6-b12a-b53dd2655045&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2966992 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2966992 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2966992 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:37910/solr/testcollection_shard1_replica_n2/
   [junit4]   2> 2966993 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=testcollection_shard1_replica_n2 url=https://127.0.0.1:37910/solr START replicas=[https://127.0.0.1:41228/solr/testcollection_shard1_replica_n1/] nUpdates=100
   [junit4]   2> 2967002 INFO  (qtp1032987856-22129) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.S.Request [testcollection_shard1_replica_n1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 2967003 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2967003 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2967003 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:37910/solr/testcollection_shard2_replica_n6/
   [junit4]   2> 2967004 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.PeerSync PeerSync: core=testcollection_shard2_replica_n6 url=https://127.0.0.1:37910/solr START replicas=[https://127.0.0.1:41228/solr/testcollection_shard2_replica_n4/] nUpdates=100
   [junit4]   2> 2967017 INFO  (qtp1032987856-22128) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.S.Request [testcollection_shard2_replica_n4]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 2967293 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=testcollection_shard1_replica_n2 url=https://127.0.0.1:37910/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 2967294 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 2967294 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 2967294 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2967298 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:37910/solr/testcollection_shard1_replica_n2/ shard1
   [junit4]   2> 2967305 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.PeerSync PeerSync: core=testcollection_shard2_replica_n6 url=https://127.0.0.1:37910/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 2967305 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 2967305 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 2967305 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2967308 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:37910/solr/testcollection_shard2_replica_n6/ shard2
   [junit4]   2> 2967344 INFO  (qtp197383304-22135) [n:127.0.0.1:37910_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048
   [junit4]   2> 2967344 INFO  (qtp197383304-22135) [n:127.0.0.1:37910_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2967411 INFO  (parallelCoreAdminExecutor-8065-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard2_replica_n6 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2967450 INFO  (parallelCoreAdminExecutor-8065-thread-2-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448297703003 CREATE) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2967556 INFO  (zkCallback-5215-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testcollection/state.json] for collection [testcollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2967676 INFO  (zkCallback-5215-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testcollection/state.json] for collection [testcollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2967676 INFO  (zkCallback-5213-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testcollection/state.json] for collection [testcollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2967945 INFO  (qtp197383304-22138) [n:127.0.0.1:37910_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=0ae7e8fb-c927-4cd6-b12a-b53dd2655045&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2967946 INFO  (qtp197383304-22138) [n:127.0.0.1:37910_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=0ae7e8fb-c927-4cd6-b12a-b53dd2655045&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 2968345 INFO  (qtp197383304-22135) [n:127.0.0.1:37910_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048
   [junit4]   2> 2968345 INFO  (qtp197383304-22135) [n:127.0.0.1:37910_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448303542048&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2968348 INFO  (qtp1032987856-22129) [n:127.0.0.1:41228_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187
   [junit4]   2> 2968348 INFO  (qtp1032987856-22129) [n:127.0.0.1:41228_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=0ae7e8fb-c927-4cd6-b12a-b53dd265504527561448300637187&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2968350 INFO  (OverseerThreadFactory-8064-thread-1) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2968951 INFO  (qtp197383304-22132) [n:127.0.0.1:37910_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=0ae7e8fb-c927-4cd6-b12a-b53dd2655045&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2968952 INFO  (OverseerCollectionConfigSetProcessor-73863861049819141-127.0.0.1:41228_solr-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2968953 INFO  (qtp197383304-22132) [n:127.0.0.1:37910_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=0ae7e8fb-c927-4cd6-b12a-b53dd2655045&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 2968954 INFO  (qtp197383304-22138) [n:127.0.0.1:37910_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :deletestatus with params requestid=0ae7e8fb-c927-4cd6-b12a-b53dd2655045&action=DELETESTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2968955 INFO  (qtp197383304-22138) [n:127.0.0.1:37910_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=0ae7e8fb-c927-4cd6-b12a-b53dd2655045&action=DELETESTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2968955 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[970938951E7E6CC1]) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: testcollection failOnTimeout:true timeout (sec):330
   [junit4]   1> -
   [junit4]   1> replica:core_node3 rstate:active live:true
   [junit4]   1> replica:core_node5 rstate:active live:true
   [junit4]   1> replica:core_node7 rstate:active live:true
   [junit4]   1> replica:core_node8 rstate:active live:true
   [junit4]   1> no one is recoverying
   [junit4]   2> 2968956 INFO  (TEST-TestAuthenticationFramework.testBasics-seed#[970938951E7E6CC1]) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: testcollection
   [junit4]   2> 2968963 INFO  (qtp1032987856-22129) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.S.Request [testcollection_shard1_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=testcollection:6&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=https://127.0.0.1:41228/solr/testcollection_shard1_replica_n1/|https://127.0.0.1:37910/solr/testcollection_shard1_replica_n2/&rows=10&version=2&q=*:*&NOW=1527309555161&isShard=true&wt=javabin} hits=0 status=0 QTime=0
   [junit4]   2> 2968963 INFO  (qtp1032987856-22126) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.S.Request [testcollection_shard2_replica_n4]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=testcollection:6&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=https://127.0.0.1:41228/solr/testcollection_shard2_replica_n4/|https://127.0.0.1:37910/solr/testcollection_shard2_replica_n6/&rows=10&version=2&q=*:*&NOW=1527309555161&isShard=true&wt=javabin} hits=0 status=0 QTime=0
   [junit4]   2> 2968964 INFO  (qtp197383304-22138) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.c.S.Request [testcollection_shard2_replica_n6]  webapp=/solr path=/select params={q=*:*&_stateVer_=testcollection:6&wt=javabin&version=2} hits=0 status=0 QTime=7
   [junit4]   2> 2969013 INFO  (qtp197383304-22132) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testcollection/terms/shard1 to Terms{values={core_node3=1, core_node5=1}, version=2}
   [junit4]   2> 2969049 INFO  (qtp1032987856-22126) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testcollection_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:37910/solr/testcollection_shard1_replica_n2/&wt=javabin&version=2}{add=[1 (1601500144127180800)]} 0 26
   [junit4]   2> 2969049 INFO  (qtp197383304-22132) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testcollection_shard1_replica_n2]  webapp=/solr path=/update params={_stateVer_=testcollection:6&wt=javabin&version=2}{add=[1 (1601500144127180800)]} 0 79
   [junit4]   2> 2969053 INFO  (qtp1032987856-22128) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1601500144214212608,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2969053 INFO  (qtp1032987856-22128) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@785b0a88 commitCommandVersion:1601500144214212608
   [junit4]   2> 2969112 INFO  (qtp197383304-22132) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1601500144276078592,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2969112 INFO  (qtp197383304-22132) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@7733be1 commitCommandVersion:1601500144276078592
   [junit4]   2> 2969117 INFO  (qtp197383304-22135) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1601500144281321472,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2969124 INFO  (qtp1032987856-22125) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1601500144288661504,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2969124 INFO  (qtp1032987856-22125) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2969124 INFO  (qtp197383304-22135) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2969126 INFO  (qtp1032987856-22128) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@785bfc6e[testcollection_shard1_replica_n1] main]
   [junit4]   2> 2969127 INFO  (qtp1032987856-22128) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2969128 INFO  (searcherExecutor-8072-thread-1-processing-n:127.0.0.1:41228_solr x:testcollection_shard1_replica_n1 c:testcollection s:shard1 r:core_node3) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.SolrCore [testcollection_shard1_replica_n1] Registered new searcher Searcher@785bfc6e[testcollection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.4.0):C1)))}
   [junit4]   2> 2969129 INFO  (qtp1032987856-22128) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testcollection_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:37910/solr/testcollection_shard1_replica_n2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 75
   [junit4]   2> 2969139 INFO  (qtp1032987856-22125) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2969139 INFO  (qtp1032987856-22125) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.u.p.LogUpdateProcessorFactory [testcollection_shard2_replica_n4]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:37910/solr/testcollection_shard1_replica_n2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 15
   [junit4]   2> 2969139 INFO  (qtp197383304-22135) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2969139 INFO  (qtp197383304-22135) [n:127.0.0.1:37910_solr c:testcollection s:shard2 r:core_node8 x:testcollection_shard2_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory [testcollection_shard2_replica_n6]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:37910/solr/testcollection_shard1_replica_n2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 22
   [junit4]   2> 2969154 INFO  (qtp197383304-22132) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@27418c12[testcollection_shard1_replica_n2] main]
   [junit4]   2> 2969154 INFO  (qtp197383304-22132) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2969156 INFO  (searcherExecutor-8070-thread-1-processing-n:127.0.0.1:37910_solr x:testcollection_shard1_replica_n2 c:testcollection s:shard1 r:core_node5) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.c.SolrCore [testcollection_shard1_replica_n2] Registered new searcher Searcher@27418c12[testcollection_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.4.0):C1)))}
   [junit4]   2> 2969157 INFO  (qtp197383304-22132) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testcollection_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:37910/solr/testcollection_shard1_replica_n2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 45
   [junit4]   2> 2969157 INFO  (qtp197383304-22138) [n:127.0.0.1:37910_solr c:testcollection s:shard1 r:core_node5 x:testcollection_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testcollection_shard1_replica_n2]  webapp=/solr path=/update params={_stateVer_=testcollection:6&commit=true&wt=javabin&version=2}{commit=} 0 104
   [junit4]   2> 2969174 INFO  (qtp1032987856-22129) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.S.Request [testcollection_shard1_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=testcollection:6&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=https://127.0.0.1:41228/solr/testcollection_shard1_replica_n1/|https://127.0.0.1:37910/solr/testcollection_shard1_replica_n2/&rows=10&version=2&q=*:*&NOW=1527309555365&isShard=true&wt=javabin} hits=1 status=0 QTime=0
   [junit4]   2> 2969176 INFO  (qtp1032987856-22125) [n:127.0.0.1:41228_solr c:testcollection s:shard2 r:core_node7 x:testcollection_shard2_replica_n4] o.a.s.c.S.Request [testcollection_shard2_replica_n4]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=testcollection:6&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=https://127.0.0.1:41228/solr/testcollection_shard2_replica_n4/|https://127.0.0.1:37910/solr/testcollection_shard2_replica_n6/&rows=10&version=2&q=*:*&NOW=1527309555365&isShard=true&wt=javabin} hits=0 status=0 QTime=0
   [junit4]   2> 2969182 INFO  (qtp1032987856-22125) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.S.Request [testcollection_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&df=text&distrib=false&_stateVer_=testcollection:6&shards.purpose=64&NOW=1527309555365&ids=1&isShard=true&shard.url=https://127.0.0.1:41228/solr/testcollection_shard1_replica_n1/|https://127.0.0.1:37910/solr/testcollection_shard1_replica_n2/&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2969183 INFO  (qtp1032987856-22126) [n:127.0.0.1:41228_solr c:testcollection s:shard1 r:core_node3 x:testcollection_shard1_replica_n1] o.a.s.c.S.Request [testcollection_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&_stateVer_=testcollection:6&wt=javabin&version=2} hits=1 status=0 QTime=23
   [junit4]   2> 2969184 INFO  (qtp197383304-22137) [n:127.0.0.1:37910_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=testcollection&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2969211 INFO  (OverseerThreadFactory-8064-thread-2) [    ] o.a.s.c.a.c.OverseerCollectionMessageHandler Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null
   [junit4]   2> 2969218 INFO  (qtp1032987856-22125) [n:127.0.0.1:41228_solr    x:testcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testcollection.shard1.replica_n1, tag=null
   [junit4]   2> 2969218 INFO  (qtp1032987856-22125) [n:127.0.0.1:41228_solr    x:testcollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@51f91d1b: rootName = solr_41228, domain = solr.core.testcollection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.testcollection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@39c7d511
   [junit4]   2> 2969240 INFO  (qtp197383304-22136) [n:127.0.0.1:37910_solr    x:testcollection_shard1_replica_n2] o.a.s.m.SolrMet

[...truncated too long message...]

d-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null1793525358
     [copy] Copying 39 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null1793525358
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null1793525358

resolve-example:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve-server:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1562408350
     [copy] Copying 246 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1562408350
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1562408350

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] 	confs: [default]
[ivy:cachepath] 	found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] 	found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] 	found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] 	found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] 	found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] 	found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] 	found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] 	found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 54ms :: artifacts dl 2ms
	---------------------------------------------------------------------
	|                  |            modules            ||   artifacts   |
	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
	---------------------------------------------------------------------
	|      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
	---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 163 minutes 17 seconds
Archiving artifacts
WARN: No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
WARN: java.lang.InterruptedException: no matches found within 10000
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)

[JENKINS] Lucene-Solr-BadApples-Tests-7.x - Build # 66 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/66/

3 tests failed.
FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node4:{"core":"c8n_1x3_lf_shard1_replica_n3","base_url":"http://127.0.0.1:44135","node_name":"127.0.0.1:44135_","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/15)={   "pullReplicas":"0",   "replicationFactor":"1",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node2":{           "core":"c8n_1x3_lf_shard1_replica_n1",           "base_url":"http://127.0.0.1:37443",           "node_name":"127.0.0.1:37443_",           "state":"down",           "type":"NRT"},         "core_node4":{           "core":"c8n_1x3_lf_shard1_replica_n3",           "base_url":"http://127.0.0.1:44135",           "node_name":"127.0.0.1:44135_",           "state":"active",           "type":"NRT",           "leader":"true"},         "core_node6":{           "core":"c8n_1x3_lf_shard1_replica_n5",           "base_url":"http://127.0.0.1:43870",           "node_name":"127.0.0.1:43870_",           "state":"down",           "type":"NRT"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"3",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node4:{"core":"c8n_1x3_lf_shard1_replica_n3","base_url":"http://127.0.0.1:44135","node_name":"127.0.0.1:44135_","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/15)={
  "pullReplicas":"0",
  "replicationFactor":"1",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"c8n_1x3_lf_shard1_replica_n1",
          "base_url":"http://127.0.0.1:37443",
          "node_name":"127.0.0.1:37443_",
          "state":"down",
          "type":"NRT"},
        "core_node4":{
          "core":"c8n_1x3_lf_shard1_replica_n3",
          "base_url":"http://127.0.0.1:44135",
          "node_name":"127.0.0.1:44135_",
          "state":"active",
          "type":"NRT",
          "leader":"true"},
        "core_node6":{
          "core":"c8n_1x3_lf_shard1_replica_n5",
          "base_url":"http://127.0.0.1:43870",
          "node_name":"127.0.0.1:43870_",
          "state":"down",
          "type":"NRT"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"3",
  "tlogReplicas":"0"}
	at __randomizedtesting.SeedInfo.seed([81E591950FFF0E5C:9B1AE4FA10363A4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:170)
	at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:57)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger

Error Message:
waitFor not elapsed but produced an event

Stack Trace:
java.lang.AssertionError: waitFor not elapsed but produced an event
	at __randomizedtesting.SeedInfo.seed([81E591950FFF0E5C:E22EA71796307D71]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger(IndexSizeTriggerTest.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.cdcr.CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap

Error Message:
Error from server at https://127.0.0.1:45544/solr: KeeperErrorCode = Session expired for /configs/cdcr-source

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:45544/solr: KeeperErrorCode = Session expired for /configs/cdcr-source
	at __randomizedtesting.SeedInfo.seed([81E591950FFF0E5C:5632BEE2BBA0961B]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1106)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.cdcr.CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap(CdcrBootstrapTest.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13575 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.IndexSizeTriggerTest_81E591950FFF0E5C-001/init-core-data-001
   [junit4]   2> 3365046 WARN  (SUITE-IndexSizeTriggerTest-seed#[81E591950FFF0E5C]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=19 numCloses=19
   [junit4]   2> 3365046 INFO  (SUITE-IndexSizeTriggerTest-seed#[81E591950FFF0E5C]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 3365047 INFO  (SUITE-IndexSizeTriggerTest-seed#[81E591950FFF0E5C]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 3365048 INFO  (SUITE-IndexSizeTriggerTest-seed#[81E591950FFF0E5C]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 3365049 INFO  (SUITE-IndexSizeTriggerTest-seed#[81E591950FFF0E5C]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.IndexSizeTriggerTest_81E591950FFF0E5C-001/tempDir-001
   [junit4]   2> 3365049 INFO  (SUITE-IndexSizeTriggerTest-seed#[81E591950FFF0E5C]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3365061 INFO  (Thread-3942) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3365061 INFO  (Thread-3942) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3365087 ERROR (Thread-3942) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3365149 INFO  (SUITE-IndexSizeTriggerTest-seed#[81E591950FFF0E5C]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:33565
   [junit4]   2> 3365164 INFO  (zkConnectionManagerCallback-6244-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3365177 INFO  (jetty-launcher-6241-thread-2) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
   [junit4]   2> 3365183 INFO  (jetty-launcher-6241-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3365183 INFO  (jetty-launcher-6241-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3365183 INFO  (jetty-launcher-6241-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3365183 INFO  (jetty-launcher-6241-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@303fb5c1{/solr,null,AVAILABLE}
   [junit4]   2> 3365184 INFO  (jetty-launcher-6241-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7dd788fd{SSL,[ssl, http/1.1]}{127.0.0.1:43848}
   [junit4]   2> 3365184 INFO  (jetty-launcher-6241-thread-2) [    ] o.e.j.s.Server Started @3365381ms
   [junit4]   2> 3365184 INFO  (jetty-launcher-6241-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43848}
   [junit4]   2> 3365185 ERROR (jetty-launcher-6241-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3365185 INFO  (jetty-launcher-6241-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3365185 INFO  (jetty-launcher-6241-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.4.0
   [junit4]   2> 3365185 INFO  (jetty-launcher-6241-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3365185 INFO  (jetty-launcher-6241-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3365185 INFO  (jetty-launcher-6241-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-05-24T14:43:25.369Z
   [junit4]   2> 3365197 INFO  (jetty-launcher-6241-thread-1) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
   [junit4]   2> 3365219 INFO  (zkConnectionManagerCallback-6246-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3365219 INFO  (jetty-launcher-6241-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3365244 INFO  (jetty-launcher-6241-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3365244 INFO  (jetty-launcher-6241-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3365244 INFO  (jetty-launcher-6241-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 3365244 INFO  (jetty-launcher-6241-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@51cad428{/solr,null,AVAILABLE}
   [junit4]   2> 3365245 INFO  (jetty-launcher-6241-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7f282e4a{SSL,[ssl, http/1.1]}{127.0.0.1:43588}
   [junit4]   2> 3365245 INFO  (jetty-launcher-6241-thread-1) [    ] o.e.j.s.Server Started @3365441ms
   [junit4]   2> 3365245 INFO  (jetty-launcher-6241-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43588}
   [junit4]   2> 3365245 INFO  (jetty-launcher-6241-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33565/solr
   [junit4]   2> 3365245 ERROR (jetty-launcher-6241-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3365245 INFO  (jetty-launcher-6241-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3365248 INFO  (jetty-launcher-6241-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.4.0
   [junit4]   2> 3365248 INFO  (jetty-launcher-6241-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3365248 INFO  (jetty-launcher-6241-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3365248 INFO  (jetty-launcher-6241-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-05-24T14:43:25.432Z
   [junit4]   2> 3365249 INFO  (zkConnectionManagerCallback-6250-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3365283 INFO  (zkConnectionManagerCallback-6252-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3365284 INFO  (jetty-launcher-6241-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3365297 INFO  (zkConnectionManagerCallback-6254-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3365317 INFO  (jetty-launcher-6241-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33565/solr
   [junit4]   2> 3365334 INFO  (zkConnectionManagerCallback-6260-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3365335 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10236dd482a0005, likely client has closed socket
   [junit4]   2> 3365354 INFO  (zkConnectionManagerCallback-6262-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3365421 INFO  (jetty-launcher-6241-thread-1) [n:127.0.0.1:43588_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3365421 INFO  (jetty-launcher-6241-thread-1) [n:127.0.0.1:43588_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:43588_solr
   [junit4]   2> 3365422 INFO  (jetty-launcher-6241-thread-1) [n:127.0.0.1:43588_solr    ] o.a.s.c.Overseer Overseer (id=72680868017733638-127.0.0.1:43588_solr-n_0000000000) starting
   [junit4]   2> 3365451 INFO  (zkConnectionManagerCallback-6269-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3365469 INFO  (jetty-launcher-6241-thread-1) [n:127.0.0.1:43588_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33565/solr ready
   [junit4]   2> 3365494 DEBUG (OverseerAutoScalingTriggerThread-72680868017733638-127.0.0.1:43588_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 3365494 INFO  (jetty-launcher-6241-thread-1) [n:127.0.0.1:43588_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43588_solr
   [junit4]   2> 3365495 DEBUG (OverseerAutoScalingTriggerThread-72680868017733638-127.0.0.1:43588_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 3365496 DEBUG (OverseerAutoScalingTriggerThread-72680868017733638-127.0.0.1:43588_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 3365496 DEBUG (OverseerAutoScalingTriggerThread-72680868017733638-127.0.0.1:43588_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 3365526 INFO  (zkCallback-6268-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3365526 INFO  (zkCallback-6261-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3365544 DEBUG (OverseerAutoScalingTriggerThread-72680868017733638-127.0.0.1:43588_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:43588_solr]
   [junit4]   2> 3365554 DEBUG (OverseerAutoScalingTriggerThread-72680868017733638-127.0.0.1:43588_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3365557 DEBUG (OverseerAutoScalingTriggerThread-72680868017733638-127.0.0.1:43588_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 3365557 DEBUG (ScheduledTrigger-4854-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 3365695 INFO  (jetty-launcher-6241-thread-2) [n:127.0.0.1:43848_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3365709 INFO  (jetty-launcher-6241-thread-2) [n:127.0.0.1:43848_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3365711 INFO  (jetty-launcher-6241-thread-2) [n:127.0.0.1:43848_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3365711 INFO  (jetty-launcher-6241-thread-2) [n:127.0.0.1:43848_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43848_solr
   [junit4]   2> 3365737 INFO  (zkCallback-6268-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3365774 INFO  (zkCallback-6261-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3365819 INFO  (zkCallback-6253-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3366462 INFO  (jetty-launcher-6241-thread-1) [n:127.0.0.1:43588_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43588.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c910b49
   [junit4]   2> 3366489 INFO  (jetty-launcher-6241-thread-1) [n:127.0.0.1:43588_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43588.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c910b49
   [junit4]   2> 3366490 INFO  (jetty-launcher-6241-thread-1) [n:127.0.0.1:43588_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43588.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c910b49
   [junit4]   2> 3366491 INFO  (jetty-launcher-6241-thread-1) [n:127.0.0.1:43588_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.IndexSizeTriggerTest_81E591950FFF0E5C-001/tempDir-001/node1/.
   [junit4]   2> 3366595 DEBUG (ScheduledTrigger-4854-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3366658 INFO  (jetty-launcher-6241-thread-2) [n:127.0.0.1:43848_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43848.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c910b49
   [junit4]   2> 3366689 INFO  (jetty-launcher-6241-thread-2) [n:127.0.0.1:43848_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43848.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c910b49
   [junit4]   2> 3366689 INFO  (jetty-launcher-6241-thread-2) [n:127.0.0.1:43848_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43848.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c910b49
   [junit4]   2> 3366690 INFO  (jetty-launcher-6241-thread-2) [n:127.0.0.1:43848_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.IndexSizeTriggerTest_81E591950FFF0E5C-001/tempDir-001/node2/.
   [junit4]   2> 3366706 INFO  (zkConnectionManagerCallback-6274-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3366707 INFO  (jetty-launcher-6241-thread-2) [n:127.0.0.1:43848_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3366708 INFO  (jetty-launcher-6241-thread-2) [n:127.0.0.1:43848_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33565/solr ready
   [junit4]   2> 3366761 INFO  (zkConnectionManagerCallback-6276-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3366763 INFO  (zkConnectionManagerCallback-6281-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3366764 INFO  (SUITE-IndexSizeTriggerTest-seed#[81E591950FFF0E5C]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3366765 INFO  (SUITE-IndexSizeTriggerTest-seed#[81E591950FFF0E5C]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33565/solr ready
   [junit4]   2> 3366786 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 3366786 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
   [junit4]   2> 3366786 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 3366786 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 3366787 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10134_solr, 127.0.0.1:10135_solr]
   [junit4]   2> 3366787 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3366787 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
   [junit4]   2> 3366787 DEBUG (ScheduledTrigger-4863-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3366808 DEBUG (ScheduledTrigger-4863-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3366829 DEBUG (ScheduledTrigger-4863-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3366849 DEBUG (ScheduledTrigger-4863-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3366869 DEBUG (ScheduledTrigger-4863-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3366889 DEBUG (ScheduledTrigger-4863-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3366909 DEBUG (ScheduledTrigger-4863-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3366930 DEBUG (ScheduledTrigger-4863-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3366950 DEBUG (ScheduledTrigger-4863-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3366970 DEBUG (ScheduledTrigger-4863-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3366991 DEBUG (ScheduledTrigger-4863-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3367018 DEBUG (ScheduledTrigger-4863-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3367051 DEBUG (ScheduledTrigger-4863-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3367079 DEBUG (ScheduledTrigger-4863-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3367116 DEBUG (ScheduledTrigger-4863-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3367169 DEBUG (ScheduledTrigger-4863-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3367173 INFO  (TEST-IndexSizeTriggerTest.testMixedBounds-seed#[81E591950FFF0E5C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMixedBounds
   [junit4]   2> 3367185 WARN  (TEST-IndexSizeTriggerTest.testMixedBounds-seed#[81E591950FFF0E5C]) [    ] o.a.s.c.a.IndexSizeTriggerTest Requires SOLR-12208
   [junit4]   2> 3367206 DEBUG (ScheduledTrigger-4863-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3367206 INFO  (TEST-IndexSizeTriggerTest.testMixedBounds-seed#[81E591950FFF0E5C]) [    ] o.a.s.c.a.IndexSizeTriggerTest #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:		2
   [junit4]   2> ## Empty nodes:	2
   [junit4]   2> ## Dead nodes:		0
   [junit4]   2> ## Collections:	[]
   [junit4]   2> ## Max replicas per node:	0
   [junit4]   2> ## Min replicas per node:	0
   [junit4]   2> ## Total replicas:		0
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> 
   [junit4]   2> 3367207 DEBUG (TEST-IndexSizeTriggerTest.testMixedBounds-seed#[81E591950FFF0E5C]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 3367209 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 3367209 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 3367210 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3367210 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 3367312 INFO  (TEST-IndexSizeTriggerTest.testMixedBounds-seed#[81E591950FFF0E5C]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testMixedBounds
   [junit4]   2> 3367548 INFO  (TEST-IndexSizeTriggerTest.testMergeIntegration-seed#[81E591950FFF0E5C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMergeIntegration
   [junit4]   2> 3367605 DEBUG (ScheduledTrigger-4854-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3367620 DEBUG (simCloudManagerPool-4862-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testMergeIntegration_collection / shard1: {"core_node1":{
   [junit4]   2>     "core":"testMergeIntegration_collection_shard1_replica_n1",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":1,
   [junit4]   2>     "node_name":"127.0.0.1:10134_solr",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "SEARCHER.searcher.numDocs":0,
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"testMergeIntegration_collection"}}
   [junit4]   2> 3367620 DEBUG (simCloudManagerPool-4862-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testMergeIntegration_collection / shard2: {"core_node3":{
   [junit4]   2>     "core":"testMergeIntegration_collection_shard2_replica_n3",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":1,
   [junit4]   2>     "node_name":"127.0.0.1:10134_solr",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "SEARCHER.searcher.numDocs":0,
   [junit4]   2>     "shard":"shard2",
   [junit4]   2>     "collection":"testMergeIntegration_collection"}}
   [junit4]   2> 3367801 DEBUG (simCloudManagerPool-4862-thread-19) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3367801 DEBUG (simCloudManagerPool-4862-thread-19) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 3367820 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 3367820 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 3367820 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3367820 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 3367823 DEBUG (simCloudManagerPool-4862-thread-20) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3367873 DEBUG (simCloudManagerPool-4862-thread-20) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 3367873 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 3367873 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 3367873 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3367873 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 3367900 DEBUG (simCloudManagerPool-4862-thread-21) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3367900 DEBUG (simCloudManagerPool-4862-thread-21) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 3367917 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 3367917 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 3367917 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3367917 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 3368101 DEBUG (ScheduledTrigger-4863-thread-2) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"2245bac94765c4Tb5sfqb9o18aysh6lkeoeqh16g",
   [junit4]   2>   "source":"index_size_trigger",
   [junit4]   2>   "eventTime":9646817751295428,
   [junit4]   2>   "eventType":"INDEXSIZE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "aboveSize":{},
   [junit4]   2>     "belowSize":{"testMergeIntegration_collection":[
   [junit4]   2>         "{\"core_node1\":{\n    \"core\":\"testMergeIntegration_collection_shard1_replica_n1\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":7,\n    \"SEARCHER.searcher.deletedDocs\":5,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":2,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":2,\n    \"__bytes__\":1,\n    \"shard\":\"shard1\",\n    \"collection\":\"testMergeIntegration_collection\"}}",
   [junit4]   2>         "{\"core_node3\":{\n    \"core\":\"testMergeIntegration_collection_shard2_replica_n3\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":3,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":3,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":3,\n    \"__bytes__\":1,\n    \"shard\":\"shard2\",\n    \"collection\":\"testMergeIntegration_collection\"}}"]},
   [junit4]   2>     "requestedOps":["Op{action=MERGESHARDS, hints={COLL_SHARD=[{\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard2\"}, {\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard1\"}]}}"]}}
   [junit4]   2> 3368102 DEBUG (ScheduledTrigger-4863-thread-2) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [index_size_trigger]
   [junit4]   2> 3368103 INFO  (ScheduledTrigger-4863-thread-2) [    ] o.a.s.c.a.IndexSizeTriggerTest =======> CapturedEvent{timestamp=9646831050838628, stage=STARTED, actionName='null', event={
   [junit4]   2>   "id":"2245bac94765c4Tb5sfqb9o18aysh6lkeoeqh16g",
   [junit4]   2>   "source":"index_size_trigger",
   [junit4]   2>   "eventTime":9646817751295428,
   [junit4]   2>   "eventType":"INDEXSIZE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "aboveSize":{},
   [junit4]   2>     "belowSize":{"testMergeIntegration_collection":[
   [junit4]   2>         "{\"core_node1\":{\n    \"core\":\"testMergeIntegration_collection_shard1_replica_n1\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":7,\n    \"SEARCHER.searcher.deletedDocs\":5,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":2,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":2,\n    \"__bytes__\":1,\n    \"shard\":\"shard1\",\n    \"collection\":\"testMergeIntegration_collection\"}}",
   [junit4]   2>         "{\"core_node3\":{\n    \"core\":\"testMergeIntegration_collection_shard2_replica_n3\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":3,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":3,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":3,\n    \"__bytes__\":1,\n    \"shard\":\"shard2\",\n    \"collection\":\"testMergeIntegration_collection\"}}"]},
   [junit4]   2>     "_enqueue_time_":9646831019567078,
   [junit4]   2>     "requestedOps":["Op{action=MERGESHARDS, hints={COLL_SHARD=[{\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard2\"}, {\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard1\"}]}}"]}}, context={}, config={
   [junit4]   2>   "trigger":"index_size_trigger",
   [junit4]   2>   "stage":[
   [junit4]   2>     "STARTED",
   [junit4]   2>     "ABORTED",
   [junit4]   2>     "SUCCEEDED",
   [junit4]   2>     "FAILED"],
   [junit4]   2>   "afterAction":[
   [junit4]   2>     "compute_plan",
   [junit4]   2>     "execute_plan"],
   [junit4]   2>   "class":"org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest$CapturingTriggerListener",
   [junit4]   2>   "beforeAction":[
   [junit4]   2>     "compute_plan",
   [junit4]   2>     "execute_plan"]}, message='null'}
   [junit4]   2> 3368113 DEBUG (AutoscalingActionExecutor-4864-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"2245bac94765c4Tb5sfqb9o18aysh6lkeoeqh16g",
   [junit4]   2>   "source":"index_size_trigger",
   [junit4]   2>   "eventTime":9646817751295428,
   [junit4]   2>   "eventType":"INDEXSIZE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "aboveSize":{},
   [junit4]   2>     "belowSize":{"testMergeIntegration_collection":[
   [junit4]   2>         "{\"core_node1\":{\n    \"core\":\"testMergeIntegration_collection_shard1_replica_n1\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":7,\n    \"SEARCHER.searcher.deletedDocs\":5,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":2,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":2,\n    \"__bytes__\":1,\n    \"shard\":\"shard1\",\n    \"collection\":\"testMergeIntegration_collection\"}}",
   [junit4]   2>         "{\"core_node3\":{\n    \"core\":\"testMergeIntegration_collection_shard2_replica_n3\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":3,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":3,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":3,\n    \"__bytes__\":1,\n    \"shard\":\"shard2\",\n    \"collection\":\"testMergeIntegration_collection\"}}"]},
   [junit4]   2>     "_enqueue_time_":9646831019567078,
   [junit4]   2>     "requestedOps":["Op{action=MERGESHARDS, hints={COLL_SHARD=[{\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard2\"}, {\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard1\"}]}}"]}}
   [junit4]   2> 3368114 INFO  (AutoscalingActionExecutor-4864-thread-1) [    ] o.a.s.c.a.IndexSizeTriggerTest =======> CapturedEvent{timestamp=9646831612048478, stage=BEFORE_ACTION, actionName='compute_plan', event={
   [junit4]   2>   "id":"2245bac94765c4Tb5sfqb9o18aysh6lkeoeqh16g",
   [junit4]   2>   "source":"index_size_trigger",
   [junit4]   2>   "eventTime":9646817751295428,
   [junit4]   2>   "eventType":"INDEXSIZE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "aboveSize":{},
   [junit4]   2>     "belowSize":{"testMergeIntegration_collection":[
   [junit4]   2>         "{\"core_node1\":{\n    \"core\":\"testMergeIntegration_collection_shard1_replica_n1\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":7,\n    \"SEARCHER.searcher.deletedDocs\":5,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":2,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":2,\n    \"__bytes__\":1,\n    \"shard\":\"shard1\",\n    \"collection\":\"testMergeIntegration_collection\"}}",
   [junit4]   2>         "{\"core_node3\":{\n    \"core\":\"testMergeIntegration_collection_shard2_replica_n3\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":3,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":3,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":3,\n    \"__bytes__\":1,\n    \"shard\":\"shard2\",\n    \"collection\":\"testMergeIntegration_collection\"}}"]},
   [junit4]   2>     "_enqueue_time_":9646831019567078,
   [junit4]   2>     "requestedOps":["Op{action=MERGESHARDS, hints={COLL_SHARD=[{\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard2\"}, {\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard1\"}]}}"]}}, context={properties.BEFORE_ACTION=[compute_plan], source=index_size_trigger}, config={
   [junit4]   2>   "trigger":"index_size_trigger",
   [junit4]   2>   "stage":[
   [junit4]   2>     "STARTED",
   [junit4]   2>     "ABORTED",
   [junit4]   2>     "SUCCEEDED",
   [junit4]   2>     "FAILED"],
   [junit4]   2>   "afterAction":[
   [junit4]   2>     "compute_plan",
   [junit4]   2>     "execute_plan"],
   [junit4]   2>   "class":"org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest$CapturingTriggerListener",
   [junit4]   2>   "beforeAction":[
   [junit4]   2>     "compute_plan",
   [junit4]   2>     "execute_plan"]}, message='null'}
   [junit4]   2> 3368114 DEBUG (AutoscalingActionExecutor-4864-thread-1) [    ] o.a.s.c.a.ComputePlanAction -- processing event: {
   [junit4]   2>   "id":"2245bac94765c4Tb5sfqb9o18aysh6lkeoeqh16g",
   [junit4]   2>   "source":"index_size_trigger",
   [junit4]   2>   "eventTime":9646817751295428,
   [junit4]   2>   "eventType":"INDEXSIZE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "aboveSize":{},
   [junit4]   2>     "belowSize":{"testMergeIntegration_collection":[
   [junit4]   2>         "{\"core_node1\":{\n    \"core\":\"testMergeIntegration_collection_shard1_replica_n1\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":7,\n    \"SEARCHER.searcher.deletedDocs\":5,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":2,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":2,\n    \"__bytes__\":1,\n    \"shard\":\"shard1\",\n    \"collection\":\"testMergeIntegration_collection\"}}",
   [junit4]   2>         "{\"core_node3\":{\n    \"core\":\"testMergeIntegration_collection_shard2_replica_n3\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":3,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":3,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":3,\n    \"__bytes__\":1,\n    \"shard\":\"shard2\",\n    \"collection\":\"testMergeIntegration_collection\"}}"]},
   [junit4]   2>     "_enqueue_time_":9646831019567078,
   [junit4]   2>     "requestedOps":["Op{action=MERGESHARDS, hints={COLL_SHARD=[{\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard2\"}, {\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard1\"}]}}"]}} with context properties: {BEFORE_ACTION=[compute_plan]}
   [junit4]   2> 3368115 INFO  (AutoscalingActionExecutor-4864-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed plan empty, remained 0 requested ops to try.
   [junit4]   2> 3368115 INFO  (AutoscalingActionExecutor-4864-thread-1) [    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 9646831659 sessionWrapper.createTime 9646831650749228, this.sessionWrapper.createTime 9646831650749228 
   [junit4]   2> 3368115 INFO  (AutoscalingActionExecutor-4864-thread-1) [    ] o.a.s.c.a.IndexSizeTriggerTest =======> CapturedEvent{timestamp=9646831661556228, stage=AFTER_ACTION, actionName='compute_plan', event={
   [junit4]   2>   "id":"2245bac94765c4Tb5sfqb9o18aysh6lkeoeqh16g",
   [junit4]   2>   "source":"index_size_trigger",
   [junit4]   2>   "eventTime":9646817751295428,
   [junit4]   2>   "eventType":"INDEXSIZE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "__start__":1,
   [junit4]   2>     "aboveSize":{},
   [junit4]   2>     "belowSize":{"testMergeIntegration_collection":[
   [junit4]   2>         "{\"core_node1\":{\n    \"core\":\"testMergeIntegration_collection_shard1_replica_n1\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":7,\n    \"SEARCHER.searcher.deletedDocs\":5,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":2,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":2,\n    \"__bytes__\":1,\n    \"shard\":\"shard1\",\n    \"collection\":\"testMergeIntegration_collection\"}}",
   [junit4]   2>         "{\"core_node3\":{\n    \"core\":\"testMergeIntegration_collection_shard2_replica_n3\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":3,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":3,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":3,\n    \"__bytes__\":1,\n    \"shard\":\"shard2\",\n    \"collection\":\"testMergeIntegration_collection\"}}"]},
   [junit4]   2>     "_enqueue_time_":9646831019567078,
   [junit4]   2>     "requestedOps":["Op{action=MERGESHARDS, hints={COLL_SHARD=[{\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard2\"}, {\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard1\"}]}}"]}}, context={properties.unsupportedOps=[Op{action=MERGESHARDS, hints={COLL_SHARD=[{
   [junit4]   2>   "first":"testMergeIntegration_collection",
   [junit4]   2>   "second":"shard2"}, {
   [junit4]   2>   "first":"testMergeIntegration_collection",
   [junit4]   2>   "second":"shard1"}]}}], properties.BEFORE_ACTION=[compute_plan], source=index_size_trigger, properties.AFTER_ACTION=[compute_plan]}, config={
   [junit4]   2>   "trigger":"index_size_trigger",
   [junit4]   2>   "stage":[
   [junit4]   2>     "STARTED",
   [junit4]   2>     "ABORTED",
   [junit4]   2>     "SUCCEEDED",
   [junit4]   2>     "FAILED"],
   [junit4]   2>   "afterAction":[
   [junit4]   2>     "compute_plan",
   [junit4]   2>     "execute_plan"],
   [junit4]   2>   "class":"org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest$CapturingTriggerListener",
   [junit4]   2>   "beforeAction":[
   [junit4]   2>     "compute_plan",
   [junit4]   2>     "execute_plan"]}, message='null'}
   [junit4]   2> 3368115 INFO  (AutoscalingActionExecutor-4864-thread-1) [    ] o.a.s.c.a.IndexSizeTriggerTest =======> CapturedEvent{timestamp=9646831677429178, stage=BEFORE_ACTION, actionName='execute_plan', event={
   [junit4]   2>   "id":"2245bac94765c4Tb5sfqb9o18aysh6lkeoeqh16g",
   [junit4]   2>   "source":"index_size_trigger",
   [junit4]   2>   "eventTime":9646817751295428,
   [junit4]   2>   "eventType":"INDEXSIZE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "__start__":1,
   [junit4]   2>     "aboveSize":{},
   [junit4]   2>     "belowSize":{"testMergeIntegration_collection":[
   [junit4]   2>         "{\"core_node1\":{\n    \"core\":\"testMergeIntegration_collection_shard1_replica_n1\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":7,\n    \"SEARCHER.searcher.deletedDocs\":5,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":2,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":2,\n    \"__bytes__\":1,\n    \"shard\":\"shard1\",\n    \"collection\":\"testMergeIntegration_collection\"}}",
   [junit4]   2>         "{\"core_node3\":{\n    \"core\":\"testMergeIntegration_collection_shard2_replica_n3\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":3,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":3,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":3,\n    \"__bytes__\":1,\n    \"shard\":\"shard2\",\n    \"collection\":\"testMergeIntegration_collection\"}}"]},
   [junit4]   2>     "_enqueue_time_":9646831019567078,
   [junit4]   2>     "requestedOps":["Op{action=MERGESHARDS, hints={COLL_SHARD=[{\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard2\"}, {\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard1\"}]}}"]}}, context={properties.unsupportedOps=[Op{action=MERGESHARDS, hints={COLL_SHARD=[{
   [junit4]   2>   "first":"testMergeIntegration_collection",
   [junit4]   2>   "second":"shard2"}, {
   [junit4]   2>   "first":"testMergeIntegration_collection",
   [junit4]   2>   "second":"shard1"}]}}], properties.BEFORE_ACTION=[compute_plan, execute_plan], source=index_size_trigger, properties.AFTER_ACTION=[compute_plan]}, config={
   [junit4]   2>   "trigger":"index_size_trigger",
   [junit4]   2>   "stage":[
   [junit4]   2>     "STARTED",
   [junit4]   2>     "ABORTED",
   [junit4]   2>     "SUCCEEDED",
   [junit4]   2>     "FAILED"],
   [junit4]   2>   "afterAction":[
   [junit4]   2>     "compute_plan",
   [junit4]   2>     "execute_plan"],
   [junit4]   2>   "class":"org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest$CapturingTriggerListener",
   [junit4]   2>   "beforeAction":[
   [junit4]   2>     "compute_plan",
   [junit4]   2>     "execute_plan"]}, message='null'}
   [junit4]   2> 3368116 DEBUG (AutoscalingActionExecutor-4864-thread-1) [    ] o.a.s.c.a.ExecutePlanAction -- processing event: {
   [junit4]   2>   "id":"2245bac94765c4Tb5sfqb9o18aysh6lkeoeqh16g",
   [junit4]   2>   "source":"index_size_trigger",
   [junit4]   2>   "eventTime":9646817751295428,
   [junit4]   2>   "eventType":"INDEXSIZE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "__start__":1,
   [junit4]   2>     "aboveSize":{},
   [junit4]   2>     "belowSize":{"testMergeIntegration_collection":[
   [junit4]   2>         "{\"core_node1\":{\n    \"core\":\"testMergeIntegration_collection_shard1_replica_n1\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":7,\n    \"SEARCHER.searcher.deletedDocs\":5,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":2,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":2,\n    \"__bytes__\":1,\n    \"shard\":\"shard1\",\n    \"collection\":\"testMergeIntegration_collection\"}}",
   [junit4]   2>         "{\"core_node3\":{\n    \"core\":\"testMergeIntegration_collection_shard2_replica_n3\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":3,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":3,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":3,\n    \"__bytes__\":1,\n    \"shard\":\"shard2\",\n    \"collection\":\"testMergeIntegration_collection\"}}"]},
   [junit4]   2>     "_enqueue_time_":9646831019567078,
   [junit4]   2>     "requestedOps":["Op{action=MERGESHARDS, hints={COLL_SHARD=[{\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard2\"}, {\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard1\"}]}}"]}} with context properties: {AFTER_ACTION=[compute_plan], unsupportedOps=[Op{action=MERGESHARDS, hints={COLL_SHARD=[{
   [junit4]   2>   "first":"testMergeIntegration_collection",
   [junit4]   2>   "second":"shard2"}, {
   [junit4]   2>   "first":"testMergeIntegration_collection",
   [junit4]   2>   "second":"shard1"}]}}], BEFORE_ACTION=[compute_plan, execute_plan]}
   [junit4]   2> 3368116 INFO  (AutoscalingActionExecutor-4864-thread-1) [    ] o.a.s.c.a.ExecutePlanAction No operations to execute for event: {
   [junit4]   2>   "id":"2245bac94765c4Tb5sfqb9o18aysh6lkeoeqh16g",
   [junit4]   2>   "source":"index_size_trigger",
   [junit4]   2>   "eventTime":9646817751295428,
   [junit4]   2>   "eventType":"INDEXSIZE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "__start__":1,
   [junit4]   2>     "aboveSize":{},
   [junit4]   2>     "belowSize":{"testMergeIntegration_collection":[
   [junit4]   2>         "{\"core_node1\":{\n    \"core\":\"testMergeIntegration_collection_shard1_replica_n1\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":7,\n    \"SEARCHER.searcher.deletedDocs\":5,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":2,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":2,\n    \"__bytes__\":1,\n    \"shard\":\"shard1\",\n    \"collection\":\"testMergeIntegration_collection\"}}",
   [junit4]   2>         "{\"core_node3\":{\n    \"core\":\"testMergeIntegration_collection_shard2_replica_n3\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":3,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":3,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":3,\n    \"__bytes__\":1,\n    \"shard\":\"shard2\",\n    \"collection\":\"testMergeIntegration_collection\"}}"]},
   [junit4]   2>     "_enqueue_time_":9646831019567078,
   [junit4]   2>     "requestedOps":["Op{action=MERGESHARDS, hints={COLL_SHARD=[{\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard2\"}, {\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard1\"}]}}"]}}
   [junit4]   2> 3368149 INFO  (AutoscalingActionExecutor-4864-thread-1) [    ] o.a.s.c.a.IndexSizeTriggerTest =======> CapturedEvent{timestamp=9646831714802128, stage=AFTER_ACTION, actionName='execute_plan', event={
   [junit4]   2>   "id":"2245bac94765c4Tb5sfqb9o18aysh6lkeoeqh16g",
   [junit4]   2>   "source":"index_size_trigger",
   [junit4]   2>   "eventTime":9646817751295428,
   [junit4]   2>   "eventType":"INDEXSIZE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "__start__":1,
   [junit4]   2>     "aboveSize":{},
   [junit4]   2>     "belowSize":{"testMergeIntegration_collection":[
   [junit4]   2>         "{\"core_node1\":{\n    \"core\":\"testMergeIntegration_collection_shard1_replica_n1\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":7,\n    \"SEARCHER.searcher.deletedDocs\":5,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":2,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":2,\n    \"__bytes__\":1,\n    \"shard\":\"shard1\",\n    \"collection\":\"testMergeIntegration_collection\"}}",
   [junit4]   2>         "{\"core_node3\":{\n    \"core\":\"testMergeIntegration_collection_shard2_replica_n3\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":3,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":3,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":3,\n    \"__bytes__\":1,\n    \"shard\":\"shard2\",\n    \"collection\":\"testMergeIntegration_collection\"}}"]},
   [junit4]   2>     "_enqueue_time_":9646831019567078,
   [junit4]   2>     "requestedOps":["Op{action=MERGESHARDS, hints={COLL_SHARD=[{\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard2\"}, {\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard1\"}]}}"]}}, context={properties.unsupportedOps=[Op{action=MERGESHARDS, hints={COLL_SHARD=[{
   [junit4]   2>   "first":"testMergeIntegration_collection",
   [junit4]   2>   "second":"shard2"}, {
   [junit4]   2>   "first":"testMergeIntegration_collection",
   [junit4]   2>   "second":"shard1"}]}}], properties.BEFORE_ACTION=[compute_plan, execute_plan], source=index_size_trigger, properties.AFTER_ACTION=[compute_plan, execute_plan]}, config={
   [junit4]   2>   "trigger":"index_size_trigger",
   [junit4]   2>   "stage":[
   [junit4]   2>     "STARTED",
   [junit4]   2>     "ABORTED",
   [junit4]   2>     "SUCCEEDED",
   [junit4]   2>     "FAILED"],
   [junit4]   2>   "afterAction":[
   [junit4]   2>     "compute_plan",
   [junit4]   2>     "execute_plan"],
   [junit4]   2>   "class":"org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest$CapturingTriggerListener",
   [junit4]   2>   "beforeAction":[
   [junit4]   2>     "compute_plan",
   [junit4]   2>     "execute_plan"]}, message='null'}
   [junit4]   2> 3368149 INFO  (AutoscalingActionExecutor-4864-thread-1) [    ] o.a.s.c.a.IndexSizeTriggerTest =======> CapturedEvent{timestamp=9646833375359778, stage=SUCCEEDED, actionName='null', event={
   [junit4]   2>   "id":"2245bac94765c4Tb5sfqb9o18aysh6lkeoeqh16g",
   [junit4]   2>   "source":"index_size_trigger",
   [junit4]   2>   "eventTime":9646817751295428,
   [junit4]   2>   "eventType":"INDEXSIZE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "__start__":1,
   [junit4]   2>     "aboveSize":{},
   [junit4]   2>     "belowSize":{"testMergeIntegration_collection":[
   [junit4]   2>         "{\"core_node1\":{\n    \"core\":\"testMergeIntegration_collection_shard1_replica_n1\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":7,\n    \"SEARCHER.searcher.deletedDocs\":5,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":2,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":2,\n    \"__bytes__\":1,\n    \"shard\":\"shard1\",\n    \"collection\":\"testMergeIntegration_collection\"}}",
   [junit4]   2>         "{\"core_node3\":{\n    \"core\":\"testMergeIntegration_collection_shard2_replica_n3\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":3,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":3,\n    \"violationType\":\"belowDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":3,\n    \"__bytes__\":1,\n    \"shard\":\"shard2\",\n    \"collection\":\"testMergeIntegration_collection\"}}"]},
   [junit4]   2>     "_enqueue_time_":9646831019567078,
   [junit4]   2>     "requestedOps":["Op{action=MERGESHARDS, hints={COLL_SHARD=[{\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard2\"}, {\n  \"first\":\"testMergeIntegration_collection\",\n  \"second\":\"shard1\"}]}}"]}}, context={}, config={
   [junit4]   2>   "trigger":"index_size_trigger",
   [junit4]   2>   "stage":[
   [junit4]   2>     "STARTED",
   [junit4]   2>     "ABORTED",
   [junit4]   2>     "SUCCEEDED",
   [junit4]   2>     "FAILED"],
   [junit4]   2>   "afterAction":[
   [junit4]   2>     "compute_plan",
   [junit4]   2>     "execute_plan"],
   [junit4]   2>   "class":"org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest$CapturingTriggerListener",
   [junit4]   2>   "beforeAction":[
   [junit4]   2>     "compute_plan",
   [junit4]   2>     "execute_plan"]}, message='null'}
   [junit4]   2> 3368151 INFO  (TEST-IndexSizeTriggerTest.testMergeIntegration-seed#[81E591950FFF0E5C]) [    ] o.a.s.c.a.IndexSizeTriggerTest #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:		2
   [junit4]   2> ## Empty nodes:	0
   [junit4]   2> ## Dead nodes:		0
   [junit4]   2> ## Collections:	[testMergeIntegration_collection]
   [junit4]   2> DocCollection(testMergeIntegration_collection//clusterstate.json/21)={
   [junit4]   2>   "replicationFactor":"2",
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"2",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"2",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "autoCreated":"true",
   [junit4]   2>   "shards":{
   [junit4]   2>     "shard2":{
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node3":{
   [junit4]   2>           "core":"testMergeIntegration_collection_shard2_replica_n3",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":3,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":1,
   [junit4]   2>           "node_name":"127.0.0.1:10134_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.numDocs":3},
   [junit4]   2>         "core_node4":{
   [junit4]   2>           "core":"testMergeIntegration_collection_shard2_replica_n4",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":3,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":1,
   [junit4]   2>           "node_name":"127.0.0.1:10135_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.numDocs":3}},
   [junit4]   2>       "range":"0-7fffffff",
   [junit4]   2>       "state":"active"},
   [junit4]   2>     "shard1":{
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node1":{
   [junit4]   2>           "core":"testMergeIntegration_collection_shard1_replica_n1",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":7,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":5,
   [junit4]   2>           "INDEX.sizeInBytes":1,
   [junit4]   2>           "node_name":"127.0.0.1:10134_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.numDocs":2},
   [junit4]   2>         "core_node2":{
   [junit4]   2>           "core":"testMergeIntegration_collection_shard1_replica_n2",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":7,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":5,
   [junit4]   2>           "INDEX.sizeInBytes":1,
   [junit4]   2>           "node_name":"127.0.0.1:10135_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.numDocs":2}},
   [junit4]   2>       "range":"80000000-ffffffff",
   [junit4]   2>       "state":"active"}}}
   [junit4]   2> ## Max replicas per node:	2
   [junit4]   2> ## Min replicas per node:	2
   [junit4]   2> ## Total replicas:		4
   [junit4]   2> ## * testMergeIntegration_collection		4
   [junit4]   2> ##		- active           4
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##		- CREATE             1
   [junit4]   2> ##		- autoscaling        3
   [junit4]   2> ##		- update            18
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: index_size_trigger
   [junit4]   2> ##		- STARTED         1
   [junit4]   2> 
   [junit4]   2> 3368151 DEBUG (TEST-IndexSizeTriggerTest.testMergeIntegration-seed#[81E591950FFF0E5C]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 3368153 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 3368153 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 3368153 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3368153 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 3368169 DEBUG (AutoscalingActionExecutor-4864-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 2809 ms for event id=2245bac94765c4Tb5sfqb9o18aysh6lkeoeqh16g
   [junit4]   2> 3368259 INFO  (TEST-IndexSizeTriggerTest.testMergeIntegration-seed#[81E591950FFF0E5C]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testMergeIntegration
   [junit4]   2> 3368642 DEBUG (ScheduledTrigger-4854-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3369650 DEBUG (ScheduledTrigger-4854-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3369703 INFO  (TEST-IndexSizeTriggerTest.testTrigger-seed#[81E591950FFF0E5C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testTrigger
   [junit4]   2> 3369763 DEBUG (simCloudManagerPool-4862-thread-35) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testTrigger_collection / shard2: {"core_node3":{
   [junit4]   2>     "core":"testTrigger_collection_shard2_replica_n3",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":1,
   [junit4]   2>     "node_name":"127.0.0.1:10134_solr",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "SEARCHER.searcher.numDocs":0,
   [junit4]   2>     "shard":"shard2",
   [junit4]   2>     "collection":"testTrigger_collection"}}
   [junit4]   2> 3369764 DEBUG (simCloudManagerPool-4862-thread-36) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testTrigger_collection / shard1: {"core_node1":{
   [junit4]   2>     "core":"testTrigger_collection_shard1_replica_n1",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":1,
   [junit4]   2>     "node_name":"127.0.0.1:10134_solr",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "SEARCHER.searcher.numDocs":0,
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"testTrigger_collection"}}
   [junit4]   2> 3369836 INFO  (TEST-IndexSizeTriggerTest.testTrigger-seed#[81E591950FFF0E5C]) [    ] o.a.s.c.a.IndexSizeTriggerTest #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:		2
   [junit4]   2> ## Empty nodes:	0
   [junit4]   2> ## Dead nodes:		0
   [junit4]   2> ## Collections:	[testTrigger_collection]
   [junit4]   2> DocCollection(testTrigger_collection//clusterstate.json/51)={
   [junit4]   2>   "replicationFactor":"2",
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"2",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"2",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "autoCreated":"true",
   [junit4]   2>   "shards":{
   [junit4]   2>     "shard2":{
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node3":{
   [junit4]   2>           "core":"testTrigger_collection_shard2_replica_n3",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":11,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":1,
   [junit4]   2>           "node_name":"127.0.0.1:10134_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.numDocs":11},
   [junit4]   2>         "core_node4":{
   [junit4]   2>           "core":"testTrigger_collection_shard2_replica_n4",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":11,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":1,
   [junit4]   2>           "node_name":"127.0.0.1:10135_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.numDocs":11}},
   [junit4]   2>       "range":"0-7fffffff",
   [junit4]   2>       "state":"active"},
   [junit4]   2>     "shard1":{
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node1":{
   [junit4]   2>           "core":"testTrigger_collection_shard1_replica_n1",
   [junit4]   2>           "leader":"true",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":14,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":1,
   [junit4]   2>           "node_name":"127.0.0.1:10134_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.numDocs":14},
   [junit4]   2>         "core_node2":{
   [junit4]   2>           "core":"testTrigger_collection_shard1_replica_n2",
   [junit4]   2>           "SEARCHER.searcher.maxDoc":14,
   [junit4]   2>           "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>           "INDEX.sizeInBytes":1,
   [junit4]   2>           "node_name":"127.0.0.1:10135_solr",
   [junit4]   2>           "state":"active",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "SEARCHER.searcher.numDocs":14}},
   [junit4]   2>       "range":"80000000-ffffffff",
   [junit4]   2>       "state":"active"}}}
   [junit4]   2> ## Max replicas per node:	2
   [junit4]   2> ## Min replicas per node:	2
   [junit4]   2> ## Total replicas:		4
   [junit4]   2> ## * testTrigger_collection		4
   [junit4]   2> ##		- active           4
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##		- CREATE             1
   [junit4]   2> ##		- update            27
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: index_size_trigger
   [junit4]   2> ##		- STARTED         1
   [junit4]   2> ##		- SUCCEEDED       1
   [junit4]   2> 
   [junit4]   2> 3369836 DEBUG (TEST-IndexSizeTriggerTest.testTrigger-seed#[81E591950FFF0E5C]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 3369836 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 3369836 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 3369836 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3369836 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 3369937 INFO  (TEST-IndexSizeTriggerTest.testTrigger-seed#[81E591950FFF0E5C]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testTrigger
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=IndexSizeTriggerTest -Dtests.method=testTrigger -Dtests.seed=81E591950FFF0E5C -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=es-CR -Dtests.timezone=Pacific/Honolulu -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 1.68s J0 | IndexSizeTriggerTest.testTrigger <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: waitFor not elapsed but produced an event
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([81E591950FFF0E5C:E22EA71796307D71]:0)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger(IndexSizeTriggerTest.java:180)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3370667 DEBUG (ScheduledTrigger-4854-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3371625 INFO  (TEST-IndexSizeTriggerTest.testSplitIntegration-seed#[81E591950FFF0E5C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSplitIntegration
   [junit4]   2> 3371667 DEBUG (ScheduledTrigger-4854-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 3371680 DEBUG (simCloudManagerPool-4862-thread-68) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testSplitIntegration_collection / shard2: {"core_node3":{
   [junit4]   2>     "core":"testSplitIntegration_collection_shard2_replica_n3",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":1,
   [junit4]   2>     "node_name":"127.0.0.1:10134_solr",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "SEARCHER.searcher.numDocs":0,
   [junit4]   2>     "shard":"shard2",
   [junit4]   2>     "collection":"testSplitIntegration_collection"}}
   [junit4]   2> 3371680 DEBUG (simCloudManagerPool-4862-thread-69) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testSplitIntegration_collection / shard1: {"core_node1":{
   [junit4]   2>     "core":"testSplitIntegration_collection_shard1_replica_n1",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":1,
   [junit4]   2>     "node_name":"127.0.0.1:10134_solr",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "SEARCHER.searcher.numDocs":0,
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"testSplitIntegration_collection"}}
   [junit4]   2> 3371693 DEBUG (simCloudManagerPool-4862-thread-70) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3371693 DEBUG (simCloudManagerPool-4862-thread-70) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 3371693 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 3371694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 3371694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3371694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 3371694 DEBUG (simCloudManagerPool-4862-thread-71) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3371694 DEBUG (simCloudManagerPool-4862-thread-71) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 3371695 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 3371695 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 3371695 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3371695 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 3371695 DEBUG (simCloudManagerPool-4862-thread-72) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 3371695 DEBUG (simCloudManagerPool-4862-thread-72) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 3371696 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 3371717 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 3371717 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 3371717 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 3371770 DEBUG (ScheduledTrigger-4863-thread-4) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"2245e7b0949ed4Tb5sfqb9o18aysh6lkeoeqh16k",
   [junit4]   2>   "source":"index_size_trigger",
   [junit4]   2>   "eventTime":9647010610454228,
   [junit4]   2>   "eventType":"INDEXSIZE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "aboveSize":{"testSplitIntegration_collection":[
   [junit4]   2>         "{\"core_node1\":{\n    \"core\":\"testSplitIntegration_collection_shard1_replica_n1\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":14,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":14,\n    \"violationType\":\"aboveDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":14,\n    \"__bytes__\":1,\n    \"shard\":\"shard1\",\n    \"collection\":\"testSplitIntegration_collection\"}}",
   [junit4]   2>         "{\"core_node3\":{\n    \"core\":\"testSplitIntegration_collection_shard2_replica_n3\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":11,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":11,\n    \"violationType\":\"aboveDocs\",\n    \"node_name\":\"127.0.0.1:10134_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"SEARCHER.searcher.numDocs\":11,\n    \"__bytes__\":1,\n    \"shard\":\"shard2\",\n    \"collection\":\"testSplitIntegration_collection\"}}"]},
   [junit4]   2>     "belowSize":{},
   [junit4]   2>     "requestedOps":[
   [junit4]   2>       "Op{action=SPLITSHARD, hints={COLL_SHARD=[{\n  \"first\":\"testSplitIntegration_collection\",\n  \"second\":\"shard1\"}]}}",
   [junit4]   2>       "Op{action=SPLITSHARD, hints={COLL_SHARD=[{\n  \"first\":\"testSplitIntegration_collection\",\n  \"second\":\"shard2\"}]}}"]}}
   [junit4]   2> 3371770 DEBUG (ScheduledTrigger-4863-thread-4) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [index_size_trigger]
   [junit4]   2> 3371770 INFO  (ScheduledTrigger-4863-thread-4) [    ] o.a.s.c.a.IndexSizeTriggerTest =======> CapturedEvent{timestamp=9647014409291328, stage=STARTED, actionName='null', event={
   [junit4]   2>   "id":"2245e7b0949ed4Tb5sfqb9o18aysh6lkeoeqh16k",
   [junit4]   2>   "source":"index_size_trigger",
   [junit4]   2>   "eventTime":9647010610454228,
   [junit4]   2>   "eventType":"INDEXSIZE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "aboveSize":{"testSplitIntegration_collection":[
   [junit4]   2>         "{\"core_node1\":{\n    \"core\":\"testSplitIntegration_collection_shard1_replica_n1\",\n    \"leader\":\"true\",\n    \"SEARCHER.searcher.maxDoc\":14,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":1,\n    \"__docs__\":14,\n    \"v

[...truncated too long message...]

ure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null1941968067
     [copy] Copying 39 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null1941968067
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null1941968067

resolve-example:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve-server:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null288666196
     [copy] Copying 246 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null288666196
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null288666196

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] 	confs: [default]
[ivy:cachepath] 	found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] 	found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] 	found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] 	found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] 	found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] 	found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] 	found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] 	found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 57ms :: artifacts dl 1ms
	---------------------------------------------------------------------
	|                  |            modules            ||   artifacts   |
	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
	---------------------------------------------------------------------
	|      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
	---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 316 minutes 23 seconds
Archiving artifacts
WARN: No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
WARN: java.lang.InterruptedException: no matches found within 10000
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)

[JENKINS] Lucene-Solr-BadApples-Tests-7.x - Build # 65 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/65/

4 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost

Error Message:
last state: DocCollection(testNodeLost//clusterstate.json/29)={   "replicationFactor":"10",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"5",   "autoAddReplicas":"false",   "nrtReplicas":"10",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard20":{       "replicas":{         "core_node191":{           "core":"testNodeLost_shard20_replica_n191",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10043_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node200":{           "core":"testNodeLost_shard20_replica_n200",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10033_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node199":{           "core":"testNodeLost_shard20_replica_n199",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10034_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node198":{           "core":"testNodeLost_shard20_replica_n198",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10078_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node197":{           "core":"testNodeLost_shard20_replica_n197",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10094_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node196":{           "core":"testNodeLost_shard20_replica_n196",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10039_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node195":{           "core":"testNodeLost_shard20_replica_n195",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10036_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node194":{           "core":"testNodeLost_shard20_replica_n194",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10015_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node193":{           "core":"testNodeLost_shard20_replica_n193",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10067_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node192":{           "core":"testNodeLost_shard20_replica_n192",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10019_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"73330000-7fffffff",       "state":"active"},     "shard10":{       "replicas":{         "core_node92":{           "core":"testNodeLost_shard10_replica_n92",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10019_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node91":{           "core":"testNodeLost_shard10_replica_n91",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10043_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node94":{           "core":"testNodeLost_shard10_replica_n94",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10015_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node93":{           "core":"testNodeLost_shard10_replica_n93",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10067_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node96":{           "core":"testNodeLost_shard10_replica_n96",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10039_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node95":{           "core":"testNodeLost_shard10_replica_n95",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10036_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node100":{           "core":"testNodeLost_shard10_replica_n100",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10033_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node98":{           "core":"testNodeLost_shard10_replica_n98",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10078_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node97":{           "core":"testNodeLost_shard10_replica_n97",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10094_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node99":{           "core":"testNodeLost_shard10_replica_n99",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10034_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"f3330000-ffffffff",       "state":"active"},     "shard11":{       "replicas":{         "core_node108":{           "core":"testNodeLost_shard11_replica_n108",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10016_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node107":{           "core":"testNodeLost_shard11_replica_n107",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10059_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node106":{           "core":"testNodeLost_shard11_replica_n106",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10097_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node105":{           "core":"testNodeLost_shard11_replica_n105",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10041_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node104":{           "core":"testNodeLost_shard11_replica_n104",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10081_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node103":{           "core":"testNodeLost_shard11_replica_n103",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10058_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node102":{           "core":"testNodeLost_shard11_replica_n102",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10077_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node101":{           "core":"testNodeLost_shard11_replica_n101",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10095_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node110":{           "core":"testNodeLost_shard11_replica_n110",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10066_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node109":{           "core":"testNodeLost_shard11_replica_n109",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10027_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"0-ccbffff",       "state":"active"},     "shard12":{       "replicas":{         "core_node119":{           "core":"testNodeLost_shard12_replica_n119",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10022_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node118":{           "core":"testNodeLost_shard12_replica_n118",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10068_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node117":{           "core":"testNodeLost_shard12_replica_n117",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10004_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node116":{           "core":"testNodeLost_shard12_replica_n116",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10046_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node115":{           "core":"testNodeLost_shard12_replica_n115",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10088_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node114":{           "core":"testNodeLost_shard12_replica_n114",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10030_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node113":{           "core":"testNodeLost_shard12_replica_n113",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10085_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node112":{           "core":"testNodeLost_shard12_replica_n112",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10099_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node111":{           "core":"testNodeLost_shard12_replica_n111",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10080_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node120":{           "core":"testNodeLost_shard12_replica_n120",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10032_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"ccc0000-1998ffff",       "state":"active"},     "shard13":{       "replicas":{         "core_node129":{           "core":"testNodeLost_shard13_replica_n129",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10023_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node128":{           "core":"testNodeLost_shard13_replica_n128",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node126":{           "core":"testNodeLost_shard13_replica_n126",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10087_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node125":{           "core":"testNodeLost_shard13_replica_n125",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10060_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node202":{           "node_name":"127.0.0.1:10072_solr",           "core":"testNodeLost_shard13_replica_n201",           "state":"active",           "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node124":{           "core":"testNodeLost_shard13_replica_n124",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10052_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node123":{           "core":"testNodeLost_shard13_replica_n123",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10079_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node122":{           "core":"testNodeLost_shard13_replica_n122",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10062_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node121":{           "core":"testNodeLost_shard13_replica_n121",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10070_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node130":{           "core":"testNodeLost_shard13_replica_n130",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10021_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"19990000-2665ffff",       "state":"active"},     "shard2":{       "replicas":{         "core_node18":{           "core":"testNodeLost_shard2_replica_n18",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10068_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node17":{           "core":"testNodeLost_shard2_replica_n17",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10004_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node19":{           "core":"testNodeLost_shard2_replica_n19",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10022_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node20":{           "core":"testNodeLost_shard2_replica_n20",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10032_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node12":{           "core":"testNodeLost_shard2_replica_n12",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10099_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node11":{           "core":"testNodeLost_shard2_replica_n11",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10080_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node14":{           "core":"testNodeLost_shard2_replica_n14",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10030_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node13":{           "core":"testNodeLost_shard2_replica_n13",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10085_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node16":{           "core":"testNodeLost_shard2_replica_n16",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10046_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node15":{           "core":"testNodeLost_shard2_replica_n15",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10088_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"8ccc0000-9998ffff",       "state":"active"},     "shard3":{       "replicas":{         "core_node29":{           "core":"testNodeLost_shard3_replica_n29",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10023_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node28":{           "core":"testNodeLost_shard3_replica_n28",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node204":{           "node_name":"127.0.0.1:10056_solr",           "core":"testNodeLost_shard3_replica_n203",           "state":"active",           "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node30":{           "core":"testNodeLost_shard3_replica_n30",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10021_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node21":{           "core":"testNodeLost_shard3_replica_n21",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10070_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node23":{           "core":"testNodeLost_shard3_replica_n23",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10079_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node22":{           "core":"testNodeLost_shard3_replica_n22",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10062_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node25":{           "core":"testNodeLost_shard3_replica_n25",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10060_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node24":{           "core":"testNodeLost_shard3_replica_n24",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10052_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node26":{           "core":"testNodeLost_shard3_replica_n26",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10087_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"99990000-a665ffff",       "state":"active"},     "shard14":{       "replicas":{         "core_node139":{           "core":"testNodeLost_shard14_replica_n139",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10002_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node138":{           "core":"testNodeLost_shard14_replica_n138",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10086_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node137":{           "core":"testNodeLost_shard14_replica_n137",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10026_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node136":{           "core":"testNodeLost_shard14_replica_n136",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10038_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node135":{           "core":"testNodeLost_shard14_replica_n135",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10025_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node134":{           "core":"testNodeLost_shard14_replica_n134",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10020_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node133":{           "core":"testNodeLost_shard14_replica_n133",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10089_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node132":{           "core":"testNodeLost_shard14_replica_n132",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10092_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node131":{           "core":"testNodeLost_shard14_replica_n131",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10075_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node140":{           "core":"testNodeLost_shard14_replica_n140",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10024_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"26660000-3332ffff",       "state":"active"},     "shard15":{       "replicas":{         "core_node149":{           "core":"testNodeLost_shard15_replica_n149",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10035_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node148":{           "core":"testNodeLost_shard15_replica_n148",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10028_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node147":{           "core":"testNodeLost_shard15_replica_n147",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10061_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node146":{           "core":"testNodeLost_shard15_replica_n146",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10005_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node145":{           "core":"testNodeLost_shard15_replica_n145",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10064_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node144":{           "core":"testNodeLost_shard15_replica_n144",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10057_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node143":{           "core":"testNodeLost_shard15_replica_n143",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10056_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node142":{           "core":"testNodeLost_shard15_replica_n142",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10010_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node141":{           "core":"testNodeLost_shard15_replica_n141",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10051_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node150":{           "core":"testNodeLost_shard15_replica_n150",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10014_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"33330000-3fffffff",       "state":"active"},     "shard4":{       "replicas":{         "core_node39":{           "core":"testNodeLost_shard4_replica_n39",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10002_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node40":{           "core":"testNodeLost_shard4_replica_n40",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10024_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node32":{           "core":"testNodeLost_shard4_replica_n32",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10092_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node31":{           "core":"testNodeLost_shard4_replica_n31",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10075_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node34":{           "core":"testNodeLost_shard4_replica_n34",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10020_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node33":{           "core":"testNodeLost_shard4_replica_n33",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10089_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node36":{           "core":"testNodeLost_shard4_replica_n36",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10038_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node35":{           "core":"testNodeLost_shard4_replica_n35",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10025_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node38":{           "core":"testNodeLost_shard4_replica_n38",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10086_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node37":{           "core":"testNodeLost_shard4_replica_n37",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10026_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"a6660000-b332ffff",       "state":"active"},     "shard5":{       "replicas":{         "core_node50":{           "core":"testNodeLost_shard5_replica_n50",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10014_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node41":{           "core":"testNodeLost_shard5_replica_n41",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10051_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node43":{           "core":"testNodeLost_shard5_replica_n43",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10056_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node42":{           "core":"testNodeLost_shard5_replica_n42",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10010_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node45":{           "core":"testNodeLost_shard5_replica_n45",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10064_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node44":{           "core":"testNodeLost_shard5_replica_n44",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10057_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node47":{           "core":"testNodeLost_shard5_replica_n47",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10061_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node46":{           "core":"testNodeLost_shard5_replica_n46",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10005_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node49":{           "core":"testNodeLost_shard5_replica_n49",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10035_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node48":{           "core":"testNodeLost_shard5_replica_n48",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10028_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"b3330000-bfffffff",       "state":"active"},     "shard16":{       "replicas":{         "core_node159":{           "core":"testNodeLost_shard16_replica_n159",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10091_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node158":{           "core":"testNodeLost_shard16_replica_n158",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10050_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node157":{           "core":"testNodeLost_shard16_replica_n157",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10018_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node156":{           "core":"testNodeLost_shard16_replica_n156",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10042_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node155":{           "core":"testNodeLost_shard16_replica_n155",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10096_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node154":{           "core":"testNodeLost_shard16_replica_n154",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10083_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node153":{           "core":"testNodeLost_shard16_replica_n153",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10072_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node152":{           "core":"testNodeLost_shard16_replica_n152",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10045_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node151":{           "core":"testNodeLost_shard16_replica_n151",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10013_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node160":{           "core":"testNodeLost_shard16_replica_n160",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10049_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"40000000-4ccbffff",       "state":"active"},     "shard17":{       "replicas":{         "core_node169":{           "core":"testNodeLost_shard17_replica_n169",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10037_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node168":{           "core":"testNodeLost_shard17_replica_n168",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10006_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node167":{           "core":"testNodeLost_shard17_replica_n167",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10084_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node166":{           "core":"testNodeLost_shard17_replica_n166",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10055_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node165":{           "core":"testNodeLost_shard17_replica_n165",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10074_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node164":{           "core":"testNodeLost_shard17_replica_n164",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10053_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node163":{           "core":"testNodeLost_shard17_replica_n163",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10007_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node162":{           "core":"testNodeLost_shard17_replica_n162",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10065_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node161":{           "core":"testNodeLost_shard17_replica_n161",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10044_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node170":{           "core":"testNodeLost_shard17_replica_n170",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10054_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"4ccc0000-5998ffff",       "state":"active"},     "shard18":{       "replicas":{         "core_node180":{           "core":"testNodeLost_shard18_replica_n180",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10029_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node179":{           "core":"testNodeLost_shard18_replica_n179",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10047_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node178":{           "core":"testNodeLost_shard18_replica_n178",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10048_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node177":{           "core":"testNodeLost_shard18_replica_n177",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10011_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node176":{           "core":"testNodeLost_shard18_replica_n176",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10098_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node175":{           "core":"testNodeLost_shard18_replica_n175",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10012_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node174":{           "core":"testNodeLost_shard18_replica_n174",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node173":{           "core":"testNodeLost_shard18_replica_n173",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10093_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node172":{           "core":"testNodeLost_shard18_replica_n172",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10076_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node171":{           "core":"testNodeLost_shard18_replica_n171",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10008_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"59990000-6665ffff",       "state":"active"},     "shard19":{       "replicas":{         "core_node190":{           "core":"testNodeLost_shard19_replica_n190",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10090_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node189":{           "core":"testNodeLost_shard19_replica_n189",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10071_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node188":{           "core":"testNodeLost_shard19_replica_n188",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10069_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node187":{           "core":"testNodeLost_shard19_replica_n187",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10073_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node186":{           "core":"testNodeLost_shard19_replica_n186",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10031_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node185":{           "core":"testNodeLost_shard19_replica_n185",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10063_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node184":{           "core":"testNodeLost_shard19_replica_n184",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10040_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node183":{           "core":"testNodeLost_shard19_replica_n183",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10009_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node182":{           "core":"testNodeLost_shard19_replica_n182",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10017_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node181":{           "core":"testNodeLost_shard19_replica_n181",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10003_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"66660000-7332ffff",       "state":"active"},     "shard1":{       "replicas":{         "core_node1":{           "core":"testNodeLost_shard1_replica_n1",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10095_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node10":{           "core":"testNodeLost_shard1_replica_n10",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10066_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node6":{           "core":"testNodeLost_shard1_replica_n6",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10097_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node7":{           "core":"testNodeLost_shard1_replica_n7",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10059_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node8":{           "core":"testNodeLost_shard1_replica_n8",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10016_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node9":{           "core":"testNodeLost_shard1_replica_n9",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10027_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node2":{           "core":"testNodeLost_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10077_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node3":{           "core":"testNodeLost_shard1_replica_n3",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10058_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node4":{           "core":"testNodeLost_shard1_replica_n4",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10081_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node5":{           "core":"testNodeLost_shard1_replica_n5",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10041_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"80000000-8ccbffff",       "state":"active"},     "shard6":{       "replicas":{         "core_node60":{           "core":"testNodeLost_shard6_replica_n60",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10049_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node52":{           "core":"testNodeLost_shard6_replica_n52",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10045_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node51":{           "core":"testNodeLost_shard6_replica_n51",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10013_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node54":{           "core":"testNodeLost_shard6_replica_n54",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10083_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node53":{           "core":"testNodeLost_shard6_replica_n53",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10072_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node56":{           "core":"testNodeLost_shard6_replica_n56",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10042_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node55":{           "core":"testNodeLost_shard6_replica_n55",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10096_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node58":{           "core":"testNodeLost_shard6_replica_n58",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10050_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node57":{           "core":"testNodeLost_shard6_replica_n57",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10018_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node59":{           "core":"testNodeLost_shard6_replica_n59",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10091_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"c0000000-cccbffff",       "state":"active"},     "shard7":{       "replicas":{         "core_node70":{           "core":"testNodeLost_shard7_replica_n70",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10054_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node61":{           "core":"testNodeLost_shard7_replica_n61",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10044_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node63":{           "core":"testNodeLost_shard7_replica_n63",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10007_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node62":{           "core":"testNodeLost_shard7_replica_n62",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10065_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node65":{           "core":"testNodeLost_shard7_replica_n65",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10074_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node64":{           "core":"testNodeLost_shard7_replica_n64",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10053_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node67":{           "core":"testNodeLost_shard7_replica_n67",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10084_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node66":{           "core":"testNodeLost_shard7_replica_n66",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10055_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node69":{           "core":"testNodeLost_shard7_replica_n69",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10037_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node68":{           "core":"testNodeLost_shard7_replica_n68",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10006_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"cccc0000-d998ffff",       "state":"active"},     "shard8":{       "replicas":{         "core_node80":{           "core":"testNodeLost_shard8_replica_n80",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10029_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node72":{           "core":"testNodeLost_shard8_replica_n72",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10076_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node71":{           "core":"testNodeLost_shard8_replica_n71",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10008_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node74":{           "core":"testNodeLost_shard8_replica_n74",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node73":{           "core":"testNodeLost_shard8_replica_n73",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10093_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node76":{           "core":"testNodeLost_shard8_replica_n76",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10098_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node75":{           "core":"testNodeLost_shard8_replica_n75",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10012_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node78":{           "core":"testNodeLost_shard8_replica_n78",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10048_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node77":{           "core":"testNodeLost_shard8_replica_n77",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10011_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node79":{           "core":"testNodeLost_shard8_replica_n79",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10047_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"d9990000-e665ffff",       "state":"active"},     "shard9":{       "replicas":{         "core_node90":{           "core":"testNodeLost_shard9_replica_n90",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10090_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node81":{           "core":"testNodeLost_shard9_replica_n81",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10003_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node83":{           "core":"testNodeLost_shard9_replica_n83",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10009_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node82":{           "core":"testNodeLost_shard9_replica_n82",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10017_solr",           "state":"down",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node85":{           "core":"testNodeLost_shard9_replica_n85",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10063_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node84":{           "core":"testNodeLost_shard9_replica_n84",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10040_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node87":{           "core":"testNodeLost_shard9_replica_n87",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10073_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node86":{           "core":"testNodeLost_shard9_replica_n86",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10031_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node89":{           "core":"testNodeLost_shard9_replica_n89",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10071_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node88":{           "core":"testNodeLost_shard9_replica_n88",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10069_solr",           "state":"active",           "type":"NRT",           "SEARCHER.searcher.numDocs":0}},       "range":"e6660000-f332ffff",       "state":"active"}}}

Stack Trace:
java.util.concurrent.TimeoutException: last state: DocCollection(testNodeLost//clusterstate.json/29)={
  "replicationFactor":"10",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"5",
  "autoAddReplicas":"false",
  "nrtReplicas":"10",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard20":{
      "replicas":{
        "core_node191":{
          "core":"testNodeLost_shard20_replica_n191",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10043_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node200":{
          "core":"testNodeLost_shard20_replica_n200",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10033_solr",
          "state":"down",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node199":{
          "core":"testNodeLost_shard20_replica_n199",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10034_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node198":{
          "core":"testNodeLost_shard20_replica_n198",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10078_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node197":{
          "core":"testNodeLost_shard20_replica_n197",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10094_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node196":{
          "core":"testNodeLost_shard20_replica_n196",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10039_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node195":{
          "core":"testNodeLost_shard20_replica_n195",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10036_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node194":{
          "core":"testNodeLost_shard20_replica_n194",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10015_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node193":{
          "core":"testNodeLost_shard20_replica_n193",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10067_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node192":{
          "core":"testNodeLost_shard20_replica_n192",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10019_solr",
          "state":"down",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0}},
      "range":"73330000-7fffffff",
      "state":"active"},
    "shard10":{
      "replicas":{
        "core_node92":{
          "core":"testNodeLost_shard10_replica_n92",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10019_solr",
          "state":"down",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node91":{
          "core":"testNodeLost_shard10_replica_n91",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10043_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node94":{
          "core":"testNodeLost_shard10_replica_n94",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10015_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node93":{
          "core":"testNodeLost_shard10_replica_n93",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10067_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node96":{
          "core":"testNodeLost_shard10_replica_n96",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10039_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node95":{
          "core":"testNodeLost_shard10_replica_n95",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10036_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node100":{
          "core":"testNodeLost_shard10_replica_n100",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10033_solr",
          "state":"down",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node98":{
          "core":"testNodeLost_shard10_replica_n98",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10078_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node97":{
          "core":"testNodeLost_shard10_replica_n97",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10094_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node99":{
          "core":"testNodeLost_shard10_replica_n99",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10034_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0}},
      "range":"f3330000-ffffffff",
      "state":"active"},
    "shard11":{
      "replicas":{
        "core_node108":{
          "core":"testNodeLost_shard11_replica_n108",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10016_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node107":{
          "core":"testNodeLost_shard11_replica_n107",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10059_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node106":{
          "core":"testNodeLost_shard11_replica_n106",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10097_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node105":{
          "core":"testNodeLost_shard11_replica_n105",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10041_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node104":{
          "core":"testNodeLost_shard11_replica_n104",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10081_solr",
          "state":"down",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node103":{
          "core":"testNodeLost_shard11_replica_n103",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10058_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node102":{
          "core":"testNodeLost_shard11_replica_n102",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10077_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node101":{
          "core":"testNodeLost_shard11_replica_n101",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10095_solr",
          "state":"down",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node110":{
          "core":"testNodeLost_shard11_replica_n110",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10066_solr",
          "state":"down",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node109":{
          "core":"testNodeLost_shard11_replica_n109",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10027_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0}},
      "range":"0-ccbffff",
      "state":"active"},
    "shard12":{
      "replicas":{
        "core_node119":{
          "core":"testNodeLost_shard12_replica_n119",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10022_solr",
          "state":"down",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node118":{
          "core":"testNodeLost_shard12_replica_n118",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10068_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node117":{
          "core":"testNodeLost_shard12_replica_n117",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10004_solr",
          "state":"down",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node116":{
          "core":"testNodeLost_shard12_replica_n116",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10046_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node115":{
          "core":"testNodeLost_shard12_replica_n115",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10088_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node114":{
          "core":"testNodeLost_shard12_replica_n114",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10030_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node113":{
          "core":"testNodeLost_shard12_replica_n113",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10085_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node112":{
          "core":"testNodeLost_shard12_replica_n112",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10099_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node111":{
          "core":"testNodeLost_shard12_replica_n111",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10080_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node120":{
          "core":"testNodeLost_shard12_replica_n120",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10032_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0}},
      "range":"ccc0000-1998ffff",
      "state":"active"},
    "shard13":{
      "replicas":{
        "core_node129":{
          "core":"testNodeLost_shard13_replica_n129",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10023_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node128":{
          "core":"testNodeLost_shard13_replica_n128",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10001_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node126":{
          "core":"testNodeLost_shard13_replica_n126",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10087_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node125":{
          "core":"testNodeLost_shard13_replica_n125",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10060_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node202":{
          "node_name":"127.0.0.1:10072_solr",
          "core":"testNodeLost_shard13_replica_n201",
          "state":"active",
          "INDEX.sizeInBytes":1,
          "type":"NRT"},
        "core_node124":{
          "core":"testNodeLost_shard13_replica_n124",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10052_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node123":{
          "core":"testNodeLost_shard13_replica_n123",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10079_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node122":{
          "core":"testNodeLost_shard13_replica_n122",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10062_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node121":{
          "core":"testNodeLost_shard13_replica_n121",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10070_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node130":{
          "core":"testNodeLost_shard13_replica_n130",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10021_solr",
          "state":"down",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0}},
      "range":"19990000-2665ffff",
      "state":"active"},
    "shard2":{
      "replicas":{
        "core_node18":{
          "core":"testNodeLost_shard2_replica_n18",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10068_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node17":{
          "core":"testNodeLost_shard2_replica_n17",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10004_solr",
          "state":"down",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node19":{
          "core":"testNodeLost_shard2_replica_n19",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10022_solr",
          "state":"down",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node20":{
          "core":"testNodeLost_shard2_replica_n20",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10032_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node12":{
          "core":"testNodeLost_shard2_replica_n12",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10099_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node11":{
          "core":"testNodeLost_shard2_replica_n11",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10080_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node14":{
          "core":"testNodeLost_shard2_replica_n14",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10030_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node13":{
          "core":"testNodeLost_shard2_replica_n13",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10085_solr",
          "state":"active",
          "type":"NRT",
    

[...truncated too long message...]

configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null4228494
     [copy] Copying 39 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null4228494
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null4228494

resolve-example:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve-server:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1679713142
     [copy] Copying 246 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1679713142
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1679713142

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] 	confs: [default]
[ivy:cachepath] 	found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] 	found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] 	found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] 	found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] 	found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] 	found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] 	found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] 	found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 64ms :: artifacts dl 2ms
	---------------------------------------------------------------------
	|                  |            modules            ||   artifacts   |
	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
	---------------------------------------------------------------------
	|      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
	---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 296 minutes 52 seconds
Archiving artifacts
WARN: No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
WARN: java.lang.InterruptedException: no matches found within 10000
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)