You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2018/11/14 19:39:18 UTC

[JENKINS] Lucene-Solr-7.x-Linux (32bit/jdk1.8.0_172) - Build # 3092 - Still Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3092/
Java: 32bit/jdk1.8.0_172 -server -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica

Error Message:
Timeout waiting for collection to become active Live Nodes: [127.0.0.1:10219_solr, 127.0.0.1:10218_solr, 127.0.0.1:10220_solr, 127.0.0.1:10217_solr, 127.0.0.1:10221_solr] Last available state: DocCollection(testCreateCollectionAddReplica//clusterstate.json/26)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   "autoCreated":"true",   "policy":"c1",   "shards":{"shard1":{       "replicas":{"core_node1":{           "core":"testCreateCollectionAddReplica_shard1_replica_n1",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10218_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"80000000-7fffffff",       "state":"active"}}}

Stack Trace:
java.lang.AssertionError: Timeout waiting for collection to become active
Live Nodes: [127.0.0.1:10219_solr, 127.0.0.1:10218_solr, 127.0.0.1:10220_solr, 127.0.0.1:10217_solr, 127.0.0.1:10221_solr]
Last available state: DocCollection(testCreateCollectionAddReplica//clusterstate.json/26)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "policy":"c1",
  "shards":{"shard1":{
      "replicas":{"core_node1":{
          "core":"testCreateCollectionAddReplica_shard1_replica_n1",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10218_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active"}}}
	at __randomizedtesting.SeedInfo.seed([2ED281EBBB84D7A:82CD4D30AAFBA5DC]:0)
	at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:70)
	at org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica(TestSimPolicyCloud.java:123)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: last ClusterState: znodeVersion: 27
live nodes:[127.0.0.1:10219_solr, 127.0.0.1:10218_solr, 127.0.0.1:10220_solr, 127.0.0.1:10217_solr, 127.0.0.1:10221_solr]
collections:{testCreateCollectionAddReplica=DocCollection(testCreateCollectionAddReplica//clusterstate.json/26)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "policy":"c1",
  "shards":{"shard1":{
      "replicas":{"core_node1":{
          "core":"testCreateCollectionAddReplica_shard1_replica_n1",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10218_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active"}}}}, last coll state: DocCollection(testCreateCollectionAddReplica//clusterstate.json/26)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "policy":"c1",
  "shards":{"shard1":{
      "replicas":{"core_node1":{
          "core":"testCreateCollectionAddReplica_shard1_replica_n1",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10218_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active"}}}
	at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:112)
	at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:64)
	... 41 more




Build Log:
[...truncated 14468 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestSimPolicyCloud_2ED281EBBB84D7A-001/init-core-data-001
   [junit4]   2> 2301476 INFO  (SUITE-TestSimPolicyCloud-seed#[2ED281EBBB84D7A]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2301476 INFO  (SUITE-TestSimPolicyCloud-seed#[2ED281EBBB84D7A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2301476 INFO  (SUITE-TestSimPolicyCloud-seed#[2ED281EBBB84D7A]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2301479 DEBUG (SUITE-TestSimPolicyCloud-seed#[2ED281EBBB84D7A]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10217_solr
   [junit4]   2> 2301479 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 2301479 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
   [junit4]   2> 2301479 DEBUG (SUITE-TestSimPolicyCloud-seed#[2ED281EBBB84D7A]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=0
   [junit4]   2> 2301479 DEBUG (SUITE-TestSimPolicyCloud-seed#[2ED281EBBB84D7A]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 2301479 INFO  (SUITE-TestSimPolicyCloud-seed#[2ED281EBBB84D7A]-worker) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2301479 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 2301479 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 2301479 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10218_solr, 127.0.0.1:10217_solr, 127.0.0.1:10221_solr, 127.0.0.1:10219_solr, 127.0.0.1:10220_solr]
   [junit4]   2> 2301480 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2301480 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
   [junit4]   2> 2301480 DEBUG (ScheduledTrigger-9723-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301490 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[2ED281EBBB84D7A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreateCollectionAddShardWithReplicaTypeUsingPolicy
   [junit4]   2> 2301490 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 2301490 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 2301490 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 2301490 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 2301490 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2301491 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 2301490 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_172]
   [junit4]   2> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_172]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 2301491 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 2301491 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 2301491 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 2301491 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 2301492 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10218_solr, 127.0.0.1:10217_solr, 127.0.0.1:10221_solr, 127.0.0.1:10219_solr, 127.0.0.1:10220_solr]
   [junit4]   2> 2301492 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2301492 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 2301492 DEBUG (ScheduledTrigger-9728-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301513 DEBUG (ScheduledTrigger-9728-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301533 DEBUG (ScheduledTrigger-9728-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301553 DEBUG (ScheduledTrigger-9728-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301573 DEBUG (ScheduledTrigger-9728-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301594 DEBUG (ScheduledTrigger-9728-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301614 DEBUG (ScheduledTrigger-9728-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301634 DEBUG (ScheduledTrigger-9728-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301654 DEBUG (ScheduledTrigger-9728-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301674 DEBUG (ScheduledTrigger-9728-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301691 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.TestSimPolicyCloud NRT 127.0.0.1:10218_solr PULL 127.0.0.1:10217_solr , TLOG 127.0.0.1:10221_solr 
   [junit4]   2> 2301692 DEBUG (simCloudManagerPool-9727-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=1
   [junit4]   2> 2301693 DEBUG (simCloudManagerPool-9727-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 1
   [junit4]   2> 2301693 DEBUG (simCloudManagerPool-9727-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 2301693 DEBUG (simCloudManagerPool-9727-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 2301693 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 2301693 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2301693 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 2301694 DEBUG (simCloudManagerPool-9727-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection policiesTest, currentVersion=2
   [junit4]   2> 2301694 DEBUG (simCloudManagerPool-9727-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=2
   [junit4]   2> 2301694 DEBUG (simCloudManagerPool-9727-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 2
   [junit4]   2> 2301694 DEBUG (ScheduledTrigger-9728-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301704 DEBUG (simCloudManagerPool-9727-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=3
   [junit4]   2> 2301704 DEBUG (simCloudManagerPool-9727-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 3
   [junit4]   2> 2301707 DEBUG (simCloudManagerPool-9727-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection policiesTest, currentVersion=4
   [junit4]   2> 2301714 DEBUG (ScheduledTrigger-9728-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301717 DEBUG (simCloudManagerPool-9727-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=4
   [junit4]   2> 2301717 DEBUG (simCloudManagerPool-9727-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 4
   [junit4]   2> 2301717 DEBUG (simCloudManagerPool-9727-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for policiesTest / s1 (currentVersion=5): {"core_node1":{
   [junit4]   2>     "core":"policiesTest_s1_replica_n1",
   [junit4]   2>     "shard":"s1",
   [junit4]   2>     "collection":"policiesTest",
   [junit4]   2>     "node_name":"127.0.0.1:10218_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 2301728 DEBUG (simCloudManagerPool-9727-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=5
   [junit4]   2> 2301728 DEBUG (simCloudManagerPool-9727-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 5
   [junit4]   2> 2301734 DEBUG (ScheduledTrigger-9728-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301738 DEBUG (simCloudManagerPool-9727-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=6
   [junit4]   2> 2301738 DEBUG (simCloudManagerPool-9727-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 6
   [junit4]   2> 2301738 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=7
   [junit4]   2> 2301738 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 7
   [junit4]   2> 2301755 DEBUG (ScheduledTrigger-9728-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301775 DEBUG (ScheduledTrigger-9728-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301795 DEBUG (ScheduledTrigger-9728-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301815 DEBUG (ScheduledTrigger-9728-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301835 DEBUG (ScheduledTrigger-9728-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301843 INFO  (simCloudManagerPool-9727-thread-7) [    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10218_solr for creating new replica of shard s3 for collection policiesTest
   [junit4]   2> 2301853 DEBUG (simCloudManagerPool-9727-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=8
   [junit4]   2> 2301853 DEBUG (simCloudManagerPool-9727-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 8
   [junit4]   2> 2301853 INFO  (simCloudManagerPool-9727-thread-7) [    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10221_solr for creating new replica of shard s3 for collection policiesTest
   [junit4]   2> 2301853 DEBUG (simCloudManagerPool-9727-thread-8) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for policiesTest / s3 (currentVersion=9): {"core_node5":{
   [junit4]   2>     "core":"policiesTest_s3_replica_n4",
   [junit4]   2>     "shard":"s3",
   [junit4]   2>     "collection":"policiesTest",
   [junit4]   2>     "node_name":"127.0.0.1:10218_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "pullReplicas":"1",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "nrtReplicas":"1",
   [junit4]   2>     "tlogReplicas":"1",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6}}
   [junit4]   2> 2301855 DEBUG (ScheduledTrigger-9728-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301864 DEBUG (simCloudManagerPool-9727-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=9
   [junit4]   2> 2301864 DEBUG (simCloudManagerPool-9727-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 9
   [junit4]   2> 2301864 INFO  (simCloudManagerPool-9727-thread-7) [    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10217_solr for creating new replica of shard s3 for collection policiesTest
   [junit4]   2> 2301874 DEBUG (simCloudManagerPool-9727-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=10
   [junit4]   2> 2301874 DEBUG (simCloudManagerPool-9727-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 10
   [junit4]   2> 2301875 DEBUG (simCloudManagerPool-9727-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=11
   [junit4]   2> 2301880 DEBUG (ScheduledTrigger-9728-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301880 DEBUG (simCloudManagerPool-9727-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 11
   [junit4]   2> 2301881 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[2ED281EBBB84D7A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreateCollectionAddShardWithReplicaTypeUsingPolicy
   [junit4]   2> 2301881 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=12
   [junit4]   2> 2301881 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 12
   [junit4]   2> 2301881 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:		5
   [junit4]   2> ## Empty nodes:	2
   [junit4]   2> ## Dead nodes:		0
   [junit4]   2> ## Collections:
   [junit4]   2> ##  * policiesTest
   [junit4]   2> ##    shardsTotal	2
   [junit4]   2> ##    shardsState	{active=2}
   [junit4]   2> ##      shardsWithoutLeader	0
   [junit4]   2> ##    totalReplicas	6
   [junit4]   2> ##      activeReplicas	6
   [junit4]   2> ##      inactiveReplicas	0
   [junit4]   2> ##    totalActiveDocs	0
   [junit4]   2> ##      bufferedDocs	0
   [junit4]   2> ##      maxActiveSliceDocs	0
   [junit4]   2> ##      minActiveSliceDocs	0
   [junit4]   2> ##      avgActiveSliceDocs	0
   [junit4]   2> ##    totalInactiveDocs	0
   [junit4]   2> ##      maxInactiveSliceDocs	0
   [junit4]   2> ##      minInactiveSliceDocs	0
   [junit4]   2> ##      avgInactiveSliceDocs	NaN
   [junit4]   2> ##    totalActiveBytes	20,480
   [junit4]   2> ##      maxActiveSliceBytes	10,240
   [junit4]   2> ##      minActiveSliceBytes	10,240
   [junit4]   2> ##      avgActiveSliceBytes	10,240
   [junit4]   2> ##    totalInactiveBytes	0
   [junit4]   2> ##      maxInactiveSliceBytes	0
   [junit4]   2> ##      minInactiveSliceBytes	0
   [junit4]   2> ##      avgInactiveSliceBytes	NaN
   [junit4]   2> ##    totalActiveDeletedDocs	0
   [junit4]   2> ## Max replicas per node:	2
   [junit4]   2> ## Min replicas per node:	0
   [junit4]   2> ## Total replicas:		6
   [junit4]   2> ## * policiesTest		6
   [junit4]   2> ##		- active           6
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##		- CREATE             1
   [junit4]   2> ##		- CREATESHARD        1
   [junit4]   2> ##		- autoscaling        1
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> 
   [junit4]   2> 2301900 INFO  (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[2ED281EBBB84D7A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDataProviderPerReplicaDetails
   [junit4]   2> 2301900 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 2301900 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 2301900 INFO  (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 2301900 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 2301900 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2301900 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 2301900 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_172]
   [junit4]   2> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_172]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 2301900 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 2301901 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 2301901 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion -1
   [junit4]   2> 2301901 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 2301901 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10218_solr, 127.0.0.1:10217_solr, 127.0.0.1:10221_solr, 127.0.0.1:10219_solr, 127.0.0.1:10220_solr]
   [junit4]   2> 2301901 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2301901 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 2301901 DEBUG (ScheduledTrigger-9731-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301922 DEBUG (ScheduledTrigger-9731-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301942 DEBUG (ScheduledTrigger-9731-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301962 DEBUG (ScheduledTrigger-9731-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2301983 DEBUG (ScheduledTrigger-9731-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302003 DEBUG (ScheduledTrigger-9731-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302023 DEBUG (ScheduledTrigger-9731-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302043 DEBUG (ScheduledTrigger-9731-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302063 DEBUG (ScheduledTrigger-9731-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302083 DEBUG (ScheduledTrigger-9731-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302101 DEBUG (simCloudManagerPool-9730-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection perReplicaDataColl, currentVersion=13
   [junit4]   2> 2302101 DEBUG (simCloudManagerPool-9730-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=13
   [junit4]   2> 2302101 DEBUG (simCloudManagerPool-9730-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 13
   [junit4]   2> 2302103 DEBUG (ScheduledTrigger-9731-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302111 DEBUG (simCloudManagerPool-9730-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=14
   [junit4]   2> 2302111 DEBUG (simCloudManagerPool-9730-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 14
   [junit4]   2> 2302113 DEBUG (simCloudManagerPool-9730-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection perReplicaDataColl, currentVersion=15
   [junit4]   2> 2302122 DEBUG (simCloudManagerPool-9730-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=15
   [junit4]   2> 2302122 DEBUG (simCloudManagerPool-9730-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 15
   [junit4]   2> 2302123 DEBUG (simCloudManagerPool-9730-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for perReplicaDataColl / shard1 (currentVersion=16): {"core_node1":{
   [junit4]   2>     "core":"perReplicaDataColl_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"perReplicaDataColl",
   [junit4]   2>     "node_name":"127.0.0.1:10220_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 2302123 DEBUG (ScheduledTrigger-9731-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302133 DEBUG (simCloudManagerPool-9730-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=16
   [junit4]   2> 2302133 DEBUG (simCloudManagerPool-9730-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 16
   [junit4]   2> 2302143 DEBUG (simCloudManagerPool-9730-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=17
   [junit4]   2> 2302143 DEBUG (simCloudManagerPool-9730-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 17
   [junit4]   2> 2302144 DEBUG (ScheduledTrigger-9731-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302154 DEBUG (simCloudManagerPool-9730-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=18
   [junit4]   2> 2302154 DEBUG (simCloudManagerPool-9730-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 18
   [junit4]   2> 2302164 DEBUG (ScheduledTrigger-9731-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302164 DEBUG (simCloudManagerPool-9730-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=19
   [junit4]   2> 2302164 DEBUG (simCloudManagerPool-9730-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 19
   [junit4]   2> 2302164 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=20
   [junit4]   2> 2302164 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 20
   [junit4]   2> 2302184 DEBUG (ScheduledTrigger-9731-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302204 DEBUG (ScheduledTrigger-9731-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302224 DEBUG (ScheduledTrigger-9731-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302244 DEBUG (ScheduledTrigger-9731-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302264 DEBUG (ScheduledTrigger-9731-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302268 INFO  (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[2ED281EBBB84D7A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDataProviderPerReplicaDetails
   [junit4]   2> 2302268 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=21
   [junit4]   2> 2302268 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 21
   [junit4]   2> 2302268 INFO  (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:		5
   [junit4]   2> ## Empty nodes:	5
   [junit4]   2> ## Dead nodes:		0
   [junit4]   2> ## Collections:
   [junit4]   2> ## Max replicas per node:	0
   [junit4]   2> ## Min replicas per node:	0
   [junit4]   2> ## Total replicas:		0
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##		- CREATE             1
   [junit4]   2> ##		- DELETE             1
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> 
   [junit4]   2> 2302285 DEBUG (ScheduledTrigger-9731-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302293 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[2ED281EBBB84D7A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreateCollectionAddReplica
   [junit4]   2> 2302293 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 2302293 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 2302293 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 2302293 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 2302293 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2302293 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 2302293 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_172]
   [junit4]   2> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_172]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 2302294 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 2302294 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 2302294 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
   [junit4]   2> 2302294 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 2302294 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10218_solr, 127.0.0.1:10217_solr, 127.0.0.1:10221_solr, 127.0.0.1:10219_solr, 127.0.0.1:10220_solr]
   [junit4]   2> 2302295 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2302295 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 2302295 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302316 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302336 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302356 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302376 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302396 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302417 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302437 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302457 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302477 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302495 DEBUG (simCloudManagerPool-9733-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=22
   [junit4]   2> 2302495 DEBUG (simCloudManagerPool-9733-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 22
   [junit4]   2> 2302495 DEBUG (simCloudManagerPool-9733-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 2302496 DEBUG (simCloudManagerPool-9733-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 2302496 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 2302496 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2302496 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 2302496 DEBUG (simCloudManagerPool-9733-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection testCreateCollectionAddReplica, currentVersion=23
   [junit4]   2> 2302496 DEBUG (simCloudManagerPool-9733-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=23
   [junit4]   2> 2302497 DEBUG (simCloudManagerPool-9733-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 23
   [junit4]   2> 2302497 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302507 DEBUG (simCloudManagerPool-9733-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=24
   [junit4]   2> 2302507 DEBUG (simCloudManagerPool-9733-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 24
   [junit4]   2> 2302508 DEBUG (simCloudManagerPool-9733-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection testCreateCollectionAddReplica, currentVersion=25
   [junit4]   2> 2302517 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302518 DEBUG (simCloudManagerPool-9733-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=25
   [junit4]   2> 2302518 DEBUG (simCloudManagerPool-9733-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 25
   [junit4]   2> 2302518 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=26
   [junit4]   2> 2302518 DEBUG (simCloudManagerPool-9733-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCreateCollectionAddReplica / shard1 (currentVersion=26): {"core_node1":{
   [junit4]   2>     "core":"testCreateCollectionAddReplica_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"testCreateCollectionAddReplica",
   [junit4]   2>     "node_name":"127.0.0.1:10218_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 2302518 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 26
   [junit4]   2> 2302537 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302558 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302578 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302598 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302618 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302638 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302658 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302678 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302699 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302719 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302739 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302759 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302779 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302799 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302819 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302839 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302860 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302880 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302900 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302920 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302940 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302960 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2302981 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303001 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303021 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303041 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303061 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303081 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303101 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303121 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303142 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303162 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303182 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303202 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303222 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303242 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303262 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303283 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303303 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303323 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303343 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303363 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303383 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303403 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303424 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303444 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303464 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303484 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303504 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303524 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303545 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303565 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303585 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303605 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303625 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303645 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303666 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303686 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303706 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303726 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303746 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303766 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303786 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303807 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303827 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303847 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303867 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303887 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303907 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303928 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303948 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303968 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2303988 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304008 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304028 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304049 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304069 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304089 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304109 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304129 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304149 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304169 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304190 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304210 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304230 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304250 DEBUG (ScheduledTrigger-9734-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304270 DEBUG (ScheduledTrigger-9734-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304290 DEBUG (ScheduledTrigger-9734-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304310 DEBUG (ScheduledTrigger-9734-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304311 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[2ED281EBBB84D7A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreateCollectionAddReplica
   [junit4]   2> 2304311 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=27
   [junit4]   2> 2304311 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 27
   [junit4]   2> 2304311 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:		5
   [junit4]   2> ## Empty nodes:	4
   [junit4]   2> ## Dead nodes:		0
   [junit4]   2> ## Collections:
   [junit4]   2> ##  * testCreateCollectionAddReplica
   [junit4]   2> ##    shardsTotal	1
   [junit4]   2> ##    shardsState	{active=1}
   [junit4]   2> ##      shardsWithoutLeader	0
   [junit4]   2> ##    totalReplicas	1
   [junit4]   2> ##      activeReplicas	1
   [junit4]   2> ##      inactiveReplicas	0
   [junit4]   2> ##    totalActiveDocs	0
   [junit4]   2> ##      bufferedDocs	0
   [junit4]   2> ##      maxActiveSliceDocs	0
   [junit4]   2> ##      minActiveSliceDocs	0
   [junit4]   2> ##      avgActiveSliceDocs	0
   [junit4]   2> ##    totalInactiveDocs	0
   [junit4]   2> ##      maxInactiveSliceDocs	0
   [junit4]   2> ##      minInactiveSliceDocs	0
   [junit4]   2> ##      avgInactiveSliceDocs	NaN
   [junit4]   2> ##    totalActiveBytes	10,240
   [junit4]   2> ##      maxActiveSliceBytes	10,240
   [junit4]   2> ##      minActiveSliceBytes	10,240
   [junit4]   2> ##      avgActiveSliceBytes	10,240
   [junit4]   2> ##    totalInactiveBytes	0
   [junit4]   2> ##      maxInactiveSliceBytes	0
   [junit4]   2> ##      minInactiveSliceBytes	0
   [junit4]   2> ##      avgInactiveSliceBytes	NaN
   [junit4]   2> ##    totalActiveDeletedDocs	0
   [junit4]   2> ## Max replicas per node:	1
   [junit4]   2> ## Min replicas per node:	0
   [junit4]   2> ## Total replicas:		1
   [junit4]   2> ## * testCreateCollectionAddReplica		1
   [junit4]   2> ##		- active           1
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##		- CREATE             1
   [junit4]   2> ##		- autoscaling        1
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSimPolicyCloud -Dtests.method=testCreateCollectionAddReplica -Dtests.seed=2ED281EBBB84D7A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=et -Dtests.timezone=America/Antigua -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 2.05s J2 | TestSimPolicyCloud.testCreateCollectionAddReplica <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Timeout waiting for collection to become active
   [junit4]    > Live Nodes: [127.0.0.1:10219_solr, 127.0.0.1:10218_solr, 127.0.0.1:10220_solr, 127.0.0.1:10217_solr, 127.0.0.1:10221_solr]
   [junit4]    > Last available state: DocCollection(testCreateCollectionAddReplica//clusterstate.json/26)={
   [junit4]    >   "replicationFactor":"1",
   [junit4]    >   "pullReplicas":"0",
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1",
   [junit4]    >   "autoAddReplicas":"false",
   [junit4]    >   "nrtReplicas":"1",
   [junit4]    >   "tlogReplicas":"0",
   [junit4]    >   "autoCreated":"true",
   [junit4]    >   "policy":"c1",
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "replicas":{"core_node1":{
   [junit4]    >           "core":"testCreateCollectionAddReplica_shard1_replica_n1",
   [junit4]    >           "SEARCHER.searcher.maxDoc":0,
   [junit4]    >           "SEARCHER.searcher.deletedDocs":0,
   [junit4]    >           "INDEX.sizeInBytes":10240,
   [junit4]    >           "node_name":"127.0.0.1:10218_solr",
   [junit4]    >           "state":"active",
   [junit4]    >           "type":"NRT",
   [junit4]    >           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]    >           "SEARCHER.searcher.numDocs":0}},
   [junit4]    >       "range":"80000000-7fffffff",
   [junit4]    >       "state":"active"}}}
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2ED281EBBB84D7A:82CD4D30AAFBA5DC]:0)
   [junit4]    > 	at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:70)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica(TestSimPolicyCloud.java:123)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: last ClusterState: znodeVersion: 27
   [junit4]    > live nodes:[127.0.0.1:10219_solr, 127.0.0.1:10218_solr, 127.0.0.1:10220_solr, 127.0.0.1:10217_solr, 127.0.0.1:10221_solr]
   [junit4]    > collections:{testCreateCollectionAddReplica=DocCollection(testCreateCollectionAddReplica//clusterstate.json/26)={
   [junit4]    >   "replicationFactor":"1",
   [junit4]    >   "pullReplicas":"0",
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1",
   [junit4]    >   "autoAddReplicas":"false",
   [junit4]    >   "nrtReplicas":"1",
   [junit4]    >   "tlogReplicas":"0",
   [junit4]    >   "autoCreated":"true",
   [junit4]    >   "policy":"c1",
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "replicas":{"core_node1":{
   [junit4]    >           "core":"testCreateCollectionAddReplica_shard1_replica_n1",
   [junit4]    >           "SEARCHER.searcher.maxDoc":0,
   [junit4]    >           "SEARCHER.searcher.deletedDocs":0,
   [junit4]    >           "INDEX.sizeInBytes":10240,
   [junit4]    >           "node_name":"127.0.0.1:10218_solr",
   [junit4]    >           "state":"active",
   [junit4]    >           "type":"NRT",
   [junit4]    >           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]    >           "SEARCHER.searcher.numDocs":0}},
   [junit4]    >       "range":"80000000-7fffffff",
   [junit4]    >       "state":"active"}}}}, last coll state: DocCollection(testCreateCollectionAddReplica//clusterstate.json/26)={
   [junit4]    >   "replicationFactor":"1",
   [junit4]    >   "pullReplicas":"0",
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1",
   [junit4]    >   "autoAddReplicas":"false",
   [junit4]    >   "nrtReplicas":"1",
   [junit4]    >   "tlogReplicas":"0",
   [junit4]    >   "autoCreated":"true",
   [junit4]    >   "policy":"c1",
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "replicas":{"core_node1":{
   [junit4]    >           "core":"testCreateCollectionAddReplica_shard1_replica_n1",
   [junit4]    >           "SEARCHER.searcher.maxDoc":0,
   [junit4]    >           "SEARCHER.searcher.deletedDocs":0,
   [junit4]    >           "INDEX.sizeInBytes":10240,
   [junit4]    >           "node_name":"127.0.0.1:10218_solr",
   [junit4]    >           "state":"active",
   [junit4]    >           "type":"NRT",
   [junit4]    >           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]    >           "SEARCHER.searcher.numDocs":0}},
   [junit4]    >       "range":"80000000-7fffffff",
   [junit4]    >       "state":"active"}}}
   [junit4]    > 	at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:112)
   [junit4]    > 	at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:64)
   [junit4]    > 	... 41 more
   [junit4] IGNOR/A 0.00s J2 | TestSimPolicyCloud.testCreateCollectionAddShardUsingPolicy
   [junit4]    > Assumption #1: 'badapple' test group is disabled (@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 2304328 INFO  (TEST-TestSimPolicyCloud.testMetricsTag-seed#[2ED281EBBB84D7A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMetricsTag
   [junit4]   2> 2304328 DEBUG (TEST-TestSimPolicyCloud.testMetricsTag-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 2304328 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 2304328 INFO  (TEST-TestSimPolicyCloud.testMetricsTag-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 2304328 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 2304328 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2304328 DEBUG (TEST-TestSimPolicyCloud.testMetricsTag-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 2304328 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_172]
   [junit4]   2> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_172]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 2304329 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 2304329 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
   [junit4]   2> 2304329 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion -1
   [junit4]   2> 2304329 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
   [junit4]   2> 2304329 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10218_solr, 127.0.0.1:10217_solr, 127.0.0.1:10221_solr, 127.0.0.1:10219_solr, 127.0.0.1:10220_solr]
   [junit4]   2> 2304329 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2304329 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
   [junit4]   2> 2304329 DEBUG (ScheduledTrigger-9737-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304351 DEBUG (ScheduledTrigger-9737-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304371 DEBUG (ScheduledTrigger-9737-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304391 DEBUG (ScheduledTrigger-9737-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304412 DEBUG (ScheduledTrigger-9737-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304432 DEBUG (ScheduledTrigger-9737-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304452 DEBUG (ScheduledTrigger-9737-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304472 DEBUG (ScheduledTrigger-9737-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304492 DEBUG (ScheduledTrigger-9737-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304512 DEBUG (ScheduledTrigger-9737-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304530 ERROR (simCloudManagerPool-9736-thread-1) [    ] o.a.s.h.RequestHandlerBase org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{set-cluster-policy=[{cores=<10, node=#ANY}, {replica=<2, shard=#EACH, node=#ANY}, {metrics:abc=overseer, replica=0}], errorMessages=[Invalid metrics: param in {
   [junit4]   2>   "metrics:abc":"overseer",
   [junit4]   2>   "replica":0} must have at 2 or 3 segments after 'metrics:' separated by ':']}], 
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.processOps(AutoScalingHandler.java:210)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:148)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:706)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:651)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 2304530 DEBUG (simCloudManagerPool-9736-thread-1) [    ] o.a.s.c.a.s.SimCloudManager -- exception handling request
   [junit4]   2> org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.processOps(AutoScalingHandler.java:210) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:148) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:706) ~[test/:?]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:651) ~[test/:?]
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 2304531 DEBUG (simCloudManagerPool-9736-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=28
   [junit4]   2> 2304531 DEBUG (simCloudManagerPool-9736-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 28
   [junit4]   2> 2304531 DEBUG (simCloudManagerPool-9736-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 2304531 DEBUG (simCloudManagerPool-9736-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
   [junit4]   2> 2304531 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
   [junit4]   2> 2304531 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2304531 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
   [junit4]   2> 2304531 DEBUG (simCloudManagerPool-9736-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection metricsTest, currentVersion=29
   [junit4]   2> 2304531 DEBUG (simCloudManagerPool-9736-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=29
   [junit4]   2> 2304531 DEBUG (simCloudManagerPool-9736-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 29
   [junit4]   2> 2304532 DEBUG (ScheduledTrigger-9737-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304541 DEBUG (simCloudManagerPool-9736-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=30
   [junit4]   2> 2304542 DEBUG (simCloudManagerPool-9736-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 30
   [junit4]   2> 2304542 DEBUG (simCloudManagerPool-9736-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection metricsTest, currentVersion=31
   [junit4]   2> 2304552 DEBUG (ScheduledTrigger-9737-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304553 DEBUG (simCloudManagerPool-9736-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=31
   [junit4]   2> 2304553 DEBUG (simCloudManagerPool-9736-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 31
   [junit4]   2> 2304553 DEBUG (TEST-TestSimPolicyCloud.testMetricsTag-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=32
   [junit4]   2> 2304553 DEBUG (TEST-TestSimPolicyCloud.testMetricsTag-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 32
   [junit4]   2> 2304553 DEBUG (simCloudManagerPool-9736-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for metricsTest / shard1 (currentVersion=32): {"core_node1":{
   [junit4]   2>     "core":"metricsTest_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"metricsTest",
   [junit4]   2>     "node_name":"127.0.0.1:10220_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 2304553 DEBUG (TEST-TestSimPolicyCloud.testMetricsTag-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=33
   [junit4]   2> 2304553 DEBUG (TEST-TestSimPolicyCloud.testMetricsTag-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 33
   [junit4]   2> 2304554 INFO  (TEST-TestSimPolicyCloud.testMetricsTag-seed#[2ED281EBBB84D7A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testMetricsTag
   [junit4]   2> 2304554 DEBUG (TEST-TestSimPolicyCloud.testMetricsTag-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=34
   [junit4]   2> 2304554 DEBUG (TEST-TestSimPolicyCloud.testMetricsTag-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 34
   [junit4]   2> 2304554 INFO  (TEST-TestSimPolicyCloud.testMetricsTag-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:		5
   [junit4]   2> ## Empty nodes:	4
   [junit4]   2> ## Dead nodes:		0
   [junit4]   2> ## Collections:
   [junit4]   2> ##  * metricsTest
   [junit4]   2> ##    shardsTotal	1
   [junit4]   2> ##    shardsState	{active=1}
   [junit4]   2> ##      shardsWithoutLeader	0
   [junit4]   2> ##    totalReplicas	1
   [junit4]   2> ##      activeReplicas	1
   [junit4]   2> ##      inactiveReplicas	0
   [junit4]   2> ##    totalActiveDocs	0
   [junit4]   2> ##      bufferedDocs	0
   [junit4]   2> ##      maxActiveSliceDocs	0
   [junit4]   2> ##      minActiveSliceDocs	0
   [junit4]   2> ##      avgActiveSliceDocs	0
   [junit4]   2> ##    totalInactiveDocs	0
   [junit4]   2> ##      maxInactiveSliceDocs	0
   [junit4]   2> ##      minInactiveSliceDocs	0
   [junit4]   2> ##      avgInactiveSliceDocs	NaN
   [junit4]   2> ##    totalActiveBytes	10,240
   [junit4]   2> ##      maxActiveSliceBytes	10,240
   [junit4]   2> ##      minActiveSliceBytes	10,240
   [junit4]   2> ##      avgActiveSliceBytes	10,240
   [junit4]   2> ##    totalInactiveBytes	0
   [junit4]   2> ##      maxInactiveSliceBytes	0
   [junit4]   2> ##      minInactiveSliceBytes	0
   [junit4]   2> ##      avgInactiveSliceBytes	NaN
   [junit4]   2> ##    totalActiveDeletedDocs	0
   [junit4]   2> ## Max replicas per node:	1
   [junit4]   2> ## Min replicas per node:	0
   [junit4]   2> ## Total replicas:		1
   [junit4]   2> ## * metricsTest		1
   [junit4]   2> ##		- active           1
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##		- CREATE             1
   [junit4]   2> ##		- autoscaling        2
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> 
   [junit4]   2> 2304573 DEBUG (ScheduledTrigger-9737-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304574 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionSplitShard-seed#[2ED281EBBB84D7A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreateCollectionSplitShard
   [junit4]   2> 2304574 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionSplitShard-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 12
   [junit4]   2> 2304574 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 12
   [junit4]   2> 2304574 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionSplitShard-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 2304574 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 2304574 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2304574 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionSplitShard-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 2304574 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_172]
   [junit4]   2> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_172]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 2304575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 2304575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 13
   [junit4]   2> 2304575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion -1
   [junit4]   2> 2304575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 13
   [junit4]   2> 2304575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10218_solr, 127.0.0.1:10217_solr, 127.0.0.1:10221_solr, 127.0.0.1:10219_solr, 127.0.0.1:10220_solr]
   [junit4]   2> 2304575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2304575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion 13
   [junit4]   2> 2304575 DEBUG (ScheduledTrigger-9740-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304597 DEBUG (ScheduledTrigger-9740-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304617 DEBUG (ScheduledTrigger-9740-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304637 DEBUG (ScheduledTrigger-9740-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304657 DEBUG (ScheduledTrigger-9740-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 2304677 DEBUG (ScheduledTrigger-9740-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Runni

[...truncated too long message...]

to_add_replicas with currently live nodes: 5
   [junit4]   2> 2305330 INFO  (TEST-TestSimPolicyCloud.testDataProvider-seed#[2ED281EBBB84D7A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDataProvider
   [junit4]   2> 2305330 DEBUG (TEST-TestSimPolicyCloud.testDataProvider-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=51
   [junit4]   2> 2305330 DEBUG (TEST-TestSimPolicyCloud.testDataProvider-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 51
   [junit4]   2> 2305331 INFO  (TEST-TestSimPolicyCloud.testDataProvider-seed#[2ED281EBBB84D7A]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:		5
   [junit4]   2> ## Empty nodes:	3
   [junit4]   2> ## Dead nodes:		0
   [junit4]   2> ## Collections:
   [junit4]   2> ##  * policiesTest
   [junit4]   2> ##    shardsTotal	1
   [junit4]   2> ##    shardsState	{active=1}
   [junit4]   2> ##      shardsWithoutLeader	0
   [junit4]   2> ##    totalReplicas	2
   [junit4]   2> ##      activeReplicas	2
   [junit4]   2> ##      inactiveReplicas	0
   [junit4]   2> ##    totalActiveDocs	0
   [junit4]   2> ##      bufferedDocs	0
   [junit4]   2> ##      maxActiveSliceDocs	0
   [junit4]   2> ##      minActiveSliceDocs	0
   [junit4]   2> ##      avgActiveSliceDocs	0
   [junit4]   2> ##    totalInactiveDocs	0
   [junit4]   2> ##      maxInactiveSliceDocs	0
   [junit4]   2> ##      minInactiveSliceDocs	0
   [junit4]   2> ##      avgInactiveSliceDocs	NaN
   [junit4]   2> ##    totalActiveBytes	10,240
   [junit4]   2> ##      maxActiveSliceBytes	10,240
   [junit4]   2> ##      minActiveSliceBytes	10,240
   [junit4]   2> ##      avgActiveSliceBytes	10,240
   [junit4]   2> ##    totalInactiveBytes	0
   [junit4]   2> ##      maxInactiveSliceBytes	0
   [junit4]   2> ##      minInactiveSliceBytes	0
   [junit4]   2> ##      avgInactiveSliceBytes	NaN
   [junit4]   2> ##    totalActiveDeletedDocs	0
   [junit4]   2> ## Max replicas per node:	1
   [junit4]   2> ## Min replicas per node:	0
   [junit4]   2> ## Total replicas:		2
   [junit4]   2> ## * policiesTest		2
   [junit4]   2> ##		- active           2
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##		- ADDROLE            1
   [junit4]   2> ##		- CREATE             1
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> 
   [junit4]   2> 2305331 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 2305331 DEBUG (SUITE-TestSimPolicyCloud-seed#[2ED281EBBB84D7A]-worker) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestSimPolicyCloud_2ED281EBBB84D7A-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1019, maxMBSortInHeap=5.941180835284276, sim=RandomSimilarity(queryNorm=false): {}, locale=et, timezone=America/Antigua
   [junit4]   2> NOTE: Linux 4.15.0-38-generic i386/Oracle Corporation 1.8.0_172 (32-bit)/cpus=8,threads=1,free=181766688,total=316653568
   [junit4]   2> NOTE: All tests run in this JVM: [DateFieldTest, TestTestInjection, TestSimNodeAddedTrigger, TestSmileRequest, UpdateRequestProcessorFactoryTest, XsltUpdateRequestHandlerTest, CleanupOldIndexTest, TriLevelCompositeIdRoutingTest, TestSchemaNameResource, UtilsToolTest, RollingRestartTest, TestExclusionRuleCollectionAccess, TestDocumentBuilder, TestSimpleTrackingShardHandler, QueryEqualityTest, TestSolrQueryParser, NodeMarkersRegistrationTest, DistributedMLTComponentTest, TestSolrConfigHandlerCloud, DistributedIntervalFacetingTest, ZookeeperStatusHandlerTest, ConnectionManagerTest, TestConfigReload, FieldMutatingUpdateProcessorTest, SpellCheckCollatorTest, PhrasesIdentificationComponentTest, TestCloudNestedDocsSort, UpdateParamsTest, TestAuthorizationFramework, TestHttpShardHandlerFactory, TestComplexPhraseLeadingWildcard, TestCollectionAPIs, ScriptEngineTest, TestManagedStopFilterFactory, TestLegacyNumericRangeQueryBuilder, ReturnFieldsTest, ChaosMonkeySafeLeaderTest, CursorMarkTest, TestDynamicURP, SuggesterWFSTTest, DefaultValueUpdateProcessorTest, TestGraphMLResponseWriter, TestLegacyNumericUtils, TestJettySolrRunner, OverriddenZkACLAndCredentialsProvidersTest, TestCloudInspectUtil, DistributedFacetExistsSmallTest, ManagedSchemaRoundRobinCloudTest, JSONWriterTest, BigEndianAscendingWordDeserializerTest, DeleteInactiveReplicaTest, TestPartialUpdateDeduplication, TestPerFieldSimilarity, TestSizeLimitedDistributedMap, CollectionsAPIAsyncDistributedZkTest, TestHalfAndHalfDocValues, AnalysisErrorHandlingTest, SimplePostToolTest, UninvertDocValuesMergePolicyTest, SimpleCollectionCreateDeleteTest, HdfsDirectoryFactoryTest, TestNumericTerms64, SpatialHeatmapFacetsTest, DirectSolrSpellCheckerTest, ParsingFieldUpdateProcessorsTest, TriggerCooldownIntegrationTest, TestSchemaVersionResource, HdfsRecoverLeaseTest, TestDistributedGrouping, AssignTest, MetricTriggerIntegrationTest, ConvertedLegacyTest, DistributedTermsComponentTest, ShardRoutingCustomTest, TestXmlQParserPlugin, ResourceLoaderTest, TestConfigSetsAPIZkFailure, TestPostingsSolrHighlighter, TestNumericRangeQuery32, TestStressUserVersions, TestReplicationHandler, TestWithCollection, RegexBytesRefFilterTest, TestSchemaManager, TestRandomCollapseQParserPlugin, TestCoreContainer, SolrMetricsIntegrationTest, TestExportWriter, V2ApiIntegrationTest, OutputWriterTest, TestStressLucene, HDFSCollectionsAPITest, TestUpdate, TestSSLRandomization, BasicZkTest, TestExceedMaxTermLength, URLClassifyProcessorTest, TestTrie, TestFacetMethods, ImplicitSnitchTest, DirectUpdateHandlerTest, SecurityConfHandlerTest, TermsComponentTest, TestMultiWordSynonyms, ChangedSchemaMergeTest, CustomCollectionTest, TestSha256AuthenticationProvider, MultiThreadedOCPTest, TestCustomSort, SolrCloudReportersTest, BadCopyFieldTest, TestTolerantUpdateProcessorCloud, TestDynamicFieldCollectionResource, TestJoin, TestSafeXMLParsing, SolrTestCaseJ4Test, TestExtendedDismaxParser, TestRemoteStreaming, TestBinaryField, SolrIndexSplitterTest, ResponseHeaderTest, ByteBuffersDirectoryFactoryTest, ScheduledTriggerIntegrationTest, SolrCmdDistributorTest, ExternalFileFieldSortTest, TestRecoveryHdfs, LeaderFailoverAfterPartitionTest, NodeAddedTriggerTest, ReplaceNodeTest, TestCustomStream, TestSimDistribStateManager, SliceStateTest, TestMiniSolrCloudClusterSSL, TestSimpleTextCodec, TestJsonFacetRefinement, TestGroupingSearch, TestHighlightDedupGrouping, TestSolrCoreProperties, TestTolerantSearch, PathHierarchyTokenizerFactoryTest, ProtectedTermFilterFactoryTest, TestCharFilters, TestLuceneMatchVersion, TestReversedWildcardFilterFactory, TestWordDelimiterFilterFactory, TestEmbeddedSolrServerAdminHandler, ConnectionReuseTest, ActionThrottleTest, AddReplicaTest, AliasIntegrationTest, BasicDistributedZkTest, CloudExitableDirectoryReaderTest, ClusterStateTest, ClusterStateUpdateTest, CollectionPropsTest, DeleteLastCustomShardedReplicaTest, DeleteNodeTest, DeleteReplicaTest, DistribJoinFromCollectionTest, DistributedQueueTest, DistributedVersionInfoTest, LeaderElectionContextKeyTest, LeaderVoteWaitTimeoutTest, LegacyCloudClusterPropTest, MetricsHistoryIntegrationTest, MoveReplicaTest, MultiSolrCloudTestCaseTest, NodeMutatorTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerModifyCollectionTest, OverseerRolesTest, OverseerTest, SolrXmlInZkTest, TestAuthenticationFramework, TestCloudConsistency, TestCloudDeleteByQuery, TestCloudPseudoReturnFields, TestCloudRecovery, TestCloudSearcherWarming, TestCryptoKeys, TestHashPartitioner, TestLeaderElectionZkExpiry, TestLeaderInitiatedRecoveryThread, TestLockTree, TestPullReplica, TestRandomRequestDistribution, TestShortCircuitedRequests, TestStressInPlaceUpdates, TestZkChroot, TlogReplayBufferedWhileIndexingTest, UnloadDistributedZkTest, VMParamsZkACLAndCredentialsProvidersTest, ZkCLITest, ZkControllerTest, ZkNodePropsTest, ZkShardTermsTest, CollectionsAPIDistributedZkTest, ScheduledMaintenanceTriggerTest, SearchRateTriggerTest, TestSimClusterStateProvider, TestSimComputePlanAction, TestSimDistributedQueue, TestSimExecutePlanAction, TestSimExtremeIndexing, TestSimNodeLostTrigger, TestSimPolicyCloud]
   [junit4] Completed [539/836 (1!)] on J2 in 3.88s, 7 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 46093 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3092/consoleText

[repro] Revision: b33027505b79669bc558f5a0884b75bc525ecaa0

[repro] Ant options: "-Dargs=-server -XX:+UseConcMarkSweepGC"
[repro] Repro line:  ant test  -Dtestcase=TestSimPolicyCloud -Dtests.method=testCreateCollectionAddReplica -Dtests.seed=2ED281EBBB84D7A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=et -Dtests.timezone=America/Antigua -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       TestSimPolicyCloud
[repro] ant compile-test

[...truncated 3580 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.TestSimPolicyCloud" -Dtests.showOutput=onerror "-Dargs=-server -XX:+UseConcMarkSweepGC" -Dtests.seed=2ED281EBBB84D7A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=et -Dtests.timezone=America/Antigua -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[...truncated 101 lines...]
[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud
[repro] Exiting with code 0

[...truncated 40 lines...]

[JENKINS-EA] Lucene-Solr-7.x-Linux (64bit/jdk-12-ea+12) - Build # 3098 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3098/
Java: 64bit/jdk-12-ea+12 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

11 tests failed.
FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards

Error Message:
Expected two shards with two replicas each null Live Nodes: [127.0.0.1:33375_solr, 127.0.0.1:34499_solr, 127.0.0.1:41379_solr, 127.0.0.1:44431_solr] Last available state: null

Stack Trace:
java.lang.AssertionError: Expected two shards with two replicas each
null
Live Nodes: [127.0.0.1:33375_solr, 127.0.0.1:34499_solr, 127.0.0.1:41379_solr, 127.0.0.1:44431_solr]
Last available state: null
	at __randomizedtesting.SeedInfo.seed([C6E92221CB29E4BC:FA7865758468B9E9]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards(DeleteReplicaTest.java:168)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards

Error Message:
Expected two shards with two replicas each null Live Nodes: [127.0.0.1:33919_solr, 127.0.0.1:34129_solr, 127.0.0.1:36285_solr, 127.0.0.1:41573_solr] Last available state: null

Stack Trace:
java.lang.AssertionError: Expected two shards with two replicas each
null
Live Nodes: [127.0.0.1:33919_solr, 127.0.0.1:34129_solr, 127.0.0.1:36285_solr, 127.0.0.1:41573_solr]
Last available state: null
	at __randomizedtesting.SeedInfo.seed([C6E92221CB29E4BC:FA7865758468B9E9]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards(DeleteReplicaTest.java:168)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards

Error Message:
Expected two shards with two replicas each null Live Nodes: [127.0.0.1:34951_solr, 127.0.0.1:35325_solr, 127.0.0.1:38479_solr, 127.0.0.1:44149_solr] Last available state: null

Stack Trace:
java.lang.AssertionError: Expected two shards with two replicas each
null
Live Nodes: [127.0.0.1:34951_solr, 127.0.0.1:35325_solr, 127.0.0.1:38479_solr, 127.0.0.1:44149_solr]
Last available state: null
	at __randomizedtesting.SeedInfo.seed([C6E92221CB29E4BC:FA7865758468B9E9]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards(DeleteReplicaTest.java:168)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards

Error Message:
Expected two shards with two replicas each null Live Nodes: [127.0.0.1:33385_solr, 127.0.0.1:35561_solr, 127.0.0.1:35655_solr, 127.0.0.1:42973_solr] Last available state: null

Stack Trace:
java.lang.AssertionError: Expected two shards with two replicas each
null
Live Nodes: [127.0.0.1:33385_solr, 127.0.0.1:35561_solr, 127.0.0.1:35655_solr, 127.0.0.1:42973_solr]
Last available state: null
	at __randomizedtesting.SeedInfo.seed([C6E92221CB29E4BC:FA7865758468B9E9]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards(DeleteReplicaTest.java:168)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCount

Error Message:
Expected a single shard with three replicas null Live Nodes: [127.0.0.1:33385_solr, 127.0.0.1:35561_solr, 127.0.0.1:35655_solr, 127.0.0.1:42973_solr] Last available state: null

Stack Trace:
java.lang.AssertionError: Expected a single shard with three replicas
null
Live Nodes: [127.0.0.1:33385_solr, 127.0.0.1:35561_solr, 127.0.0.1:35655_solr, 127.0.0.1:42973_solr]
Last available state: null
	at __randomizedtesting.SeedInfo.seed([C6E92221CB29E4BC:AA4CE6557C3FD1C8]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCount(DeleteReplicaTest.java:143)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.TestUtilizeNode.test

Error Message:
jettyY should now be utilized:  127.0.0.1:44057_solr => []

Stack Trace:
java.lang.AssertionError: jettyY should now be utilized:  127.0.0.1:44057_solr => []
	at __randomizedtesting.SeedInfo.seed([C6E92221CB29E4BC:4EBD1DFB65D58944]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.TestUtilizeNode.assertSomeReplicas(TestUtilizeNode.java:158)
	at org.apache.solr.cloud.TestUtilizeNode.test(TestUtilizeNode.java:133)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.TestUtilizeNode.test

Error Message:
jettyY should now be utilized:  127.0.0.1:33931_solr => []

Stack Trace:
java.lang.AssertionError: jettyY should now be utilized:  127.0.0.1:33931_solr => []
	at __randomizedtesting.SeedInfo.seed([C6E92221CB29E4BC:4EBD1DFB65D58944]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.TestUtilizeNode.assertSomeReplicas(TestUtilizeNode.java:158)
	at org.apache.solr.cloud.TestUtilizeNode.test(TestUtilizeNode.java:133)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.TestUtilizeNode.test

Error Message:
jettyY should now be utilized:  127.0.0.1:34581_solr => []

Stack Trace:
java.lang.AssertionError: jettyY should now be utilized:  127.0.0.1:34581_solr => []
	at __randomizedtesting.SeedInfo.seed([C6E92221CB29E4BC:4EBD1DFB65D58944]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.TestUtilizeNode.assertSomeReplicas(TestUtilizeNode.java:158)
	at org.apache.solr.cloud.TestUtilizeNode.test(TestUtilizeNode.java:133)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.TestUtilizeNode.test

Error Message:
jettyY should now be utilized:  127.0.0.1:37305_solr => []

Stack Trace:
java.lang.AssertionError: jettyY should now be utilized:  127.0.0.1:37305_solr => []
	at __randomizedtesting.SeedInfo.seed([C6E92221CB29E4BC:4EBD1DFB65D58944]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.TestUtilizeNode.assertSomeReplicas(TestUtilizeNode.java:158)
	at org.apache.solr.cloud.TestUtilizeNode.test(TestUtilizeNode.java:133)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.TestUtilizeNode.test

Error Message:
jettyY should now be utilized:  127.0.0.1:36855_solr => []

Stack Trace:
java.lang.AssertionError: jettyY should now be utilized:  127.0.0.1:36855_solr => []
	at __randomizedtesting.SeedInfo.seed([C6E92221CB29E4BC:4EBD1DFB65D58944]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.TestUtilizeNode.assertSomeReplicas(TestUtilizeNode.java:158)
	at org.apache.solr.cloud.TestUtilizeNode.test(TestUtilizeNode.java:133)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaFromClusterState

Error Message:
Collection not found: deleteFromClusterState_true

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: deleteFromClusterState_true
	at __randomizedtesting.SeedInfo.seed([C6E92221CB29E4BC:2870894CF4179F0B]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:851)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:177)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaFromClusterState(DeleteReplicaTest.java:189)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaFromClusterState(DeleteReplicaTest.java:179)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)




Build Log:
[...truncated 1911 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/core/test/temp/junit4-J2-20181115_172717_3162018309853040902903.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/core/test/temp/junit4-J0-20181115_172717_3168293490970381979606.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/core/test/temp/junit4-J1-20181115_172717_3169383455538913381349.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 304 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/test-framework/test/temp/junit4-J2-20181115_173433_1145960469512155076734.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/test-framework/test/temp/junit4-J0-20181115_173433_11411582463532239468605.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/test-framework/test/temp/junit4-J1-20181115_173433_11412277899682856389424.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 1080 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/common/test/temp/junit4-J0-20181115_173551_1604848238883287441752.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/common/test/temp/junit4-J1-20181115_173551_1602824468081520652182.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/common/test/temp/junit4-J2-20181115_173551_16117758995390991722362.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 255 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/icu/test/temp/junit4-J1-20181115_173739_6841357938126592141484.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/icu/test/temp/junit4-J2-20181115_173739_6885723134974538059074.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/icu/test/temp/junit4-J0-20181115_173739_6805634697473850904105.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 256 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/kuromoji/test/temp/junit4-J1-20181115_173752_5695894550905488649014.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/kuromoji/test/temp/junit4-J0-20181115_173752_56912714189490106544353.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/kuromoji/test/temp/junit4-J2-20181115_173752_5749392985174643048684.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 162 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/morfologik/test/temp/junit4-J1-20181115_173822_28116405992731813856293.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/morfologik/test/temp/junit4-J0-20181115_173822_28115915695275002128905.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/morfologik/test/temp/junit4-J2-20181115_173822_2812337907801415546427.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 207 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/nori/test/temp/junit4-J2-20181115_173826_87817960682831799181121.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/nori/test/temp/junit4-J0-20181115_173826_8789944829913604499251.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/nori/test/temp/junit4-J1-20181115_173826_8783707401217587561674.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 174 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/opennlp/test/temp/junit4-J2-20181115_173837_12917036739228444327170.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/opennlp/test/temp/junit4-J1-20181115_173837_12916914211567971064898.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/opennlp/test/temp/junit4-J0-20181115_173837_12914452042585238305014.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 176 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/phonetic/test/temp/junit4-J2-20181115_173840_41813354192311883061759.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/phonetic/test/temp/junit4-J1-20181115_173840_41810748367898855515342.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/phonetic/test/temp/junit4-J0-20181115_173840_4188076530349344224953.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 161 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/smartcn/test/temp/junit4-J0-20181115_173853_6529007320091382735280.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/smartcn/test/temp/junit4-J1-20181115_173853_6528879873074705194945.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 162 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/stempel/test/temp/junit4-J1-20181115_173900_665933868216324121423.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/stempel/test/temp/junit4-J2-20181115_173900_66512853466758552367423.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/stempel/test/temp/junit4-J0-20181115_173900_66518042533560969421121.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 206 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/backward-codecs/test/temp/junit4-J1-20181115_173903_82115982731330799159237.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 19 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/backward-codecs/test/temp/junit4-J2-20181115_173903_82111447197868810458030.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 9 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/backward-codecs/test/temp/junit4-J0-20181115_173903_8212806586888269770074.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 1549 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/benchmark/test/temp/junit4-J1-20181115_174103_7974763888541514688048.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/benchmark/test/temp/junit4-J2-20181115_174103_80112416098489938766701.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 9 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/benchmark/test/temp/junit4-J0-20181115_174103_79711064608731497004305.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 250 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/classification/test/temp/junit4-J1-20181115_174112_87912795788067682995876.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/classification/test/temp/junit4-J2-20181115_174112_8834987091467745708262.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/classification/test/temp/junit4-J0-20181115_174112_87913369756493600211673.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 258 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/codecs/test/temp/junit4-J1-20181115_174123_75210295132014489991201.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 12 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/codecs/test/temp/junit4-J0-20181115_174123_75216954628349258247662.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 4 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/codecs/test/temp/junit4-J2-20181115_174123_75216135561334064104986.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 236 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/demo/test/temp/junit4-J1-20181115_174440_5918162317329407450525.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/demo/test/temp/junit4-J0-20181115_174440_589643325537365130173.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/demo/test/temp/junit4-J2-20181115_174440_59116031919529241944720.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 176 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/expressions/test/temp/junit4-J1-20181115_174443_86414718401609471432654.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/expressions/test/temp/junit4-J0-20181115_174443_86411914964448272926382.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/expressions/test/temp/junit4-J2-20181115_174443_86410041391935067040545.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 234 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/facet/test/temp/junit4-J1-20181115_174449_70815005249491128977368.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/facet/test/temp/junit4-J0-20181115_174449_70818218129582541456406.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/facet/test/temp/junit4-J2-20181115_174449_70817017220970198850219.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 182 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/grouping/test/temp/junit4-J0-20181115_174520_71214357316617331517386.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/grouping/test/temp/junit4-J1-20181115_174520_71214782614281229247986.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/grouping/test/temp/junit4-J2-20181115_174520_7122788702041772626670.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 236 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/highlighter/test/temp/junit4-J0-20181115_174530_27415122112257441646735.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/highlighter/test/temp/junit4-J1-20181115_174530_2749310605793742534184.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 15 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/highlighter/test/temp/junit4-J2-20181115_174530_2742540943881318119742.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 166 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/join/test/temp/junit4-J1-20181115_174555_0146090222621455313998.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/join/test/temp/junit4-J2-20181115_174555_0144800723093886445575.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/join/test/temp/junit4-J0-20181115_174555_01411881370630922603825.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 157 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/memory/test/temp/junit4-J0-20181115_174623_85215101280903630126145.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/memory/test/temp/junit4-J1-20181115_174623_8527889897334861425559.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 186 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/misc/test/temp/junit4-J1-20181115_174633_6131411920288378511619.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/misc/test/temp/junit4-J0-20181115_174633_6137738269585956607800.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/misc/test/temp/junit4-J2-20181115_174633_6132833852737482412246.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 338 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/queries/test/temp/junit4-J0-20181115_174649_4177150732716385620404.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/queries/test/temp/junit4-J1-20181115_174649_4179914743702649235661.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/queries/test/temp/junit4-J2-20181115_174649_4187458684674189094399.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 231 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/queryparser/test/temp/junit4-J0-20181115_174657_6189100456511714822150.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/queryparser/test/temp/junit4-J2-20181115_174657_6186968456744992918530.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/queryparser/test/temp/junit4-J1-20181115_174657_6188292576216277055855.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 216 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/replicator/test/temp/junit4-J1-20181115_174704_96915710010473976763828.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/replicator/test/temp/junit4-J2-20181115_174704_96913631203041136252615.syserr
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/replicator/test/temp/junit4-J0-20181115_174704_9699595411059239704178.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 215 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/sandbox/test/temp/junit4-J0-20181115_174717_37916749462931650891385.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/sandbox/test/temp/junit4-J2-20181115_174717_3794885811389045639687.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/sandbox/test/temp/junit4-J1-20181115_174717_3794001400802102547033.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 299 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/spatial-extras/test/temp/junit4-J0-20181115_175110_90215877465194048942547.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/spatial-extras/test/temp/junit4-J2-20181115_175110_90216144570574522585497.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/spatial-extras/test/temp/junit4-J1-20181115_175110_9028873537818689653106.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 153 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/spatial/test/temp/junit4-J0-20181115_175130_58611199637692984542433.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 184 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/spatial3d/test/temp/junit4-J0-20181115_175132_6225382218212222127274.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/spatial3d/test/temp/junit4-J2-20181115_175132_6227916565624974991024.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/spatial3d/test/temp/junit4-J1-20181115_175132_62214828982538127444358.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 258 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/suggest/test/temp/junit4-J2-20181115_175325_1259342329655970559029.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/suggest/test/temp/junit4-J1-20181115_175325_1259469729020489015734.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/suggest/test/temp/junit4-J0-20181115_175325_1259114689318961696346.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3002 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_C6E92221CB29E4BC-001/init-core-data-001
   [junit4]   2> 781656 WARN  (SUITE-DeleteReplicaTest-seed#[C6E92221CB29E4BC]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=61 numCloses=61
   [junit4]   2> 781656 INFO  (SUITE-DeleteReplicaTest-seed#[C6E9

[...truncated too long message...]

x100644e60800012, likely client has closed socket
   [junit4]   2> 388929 INFO  (jetty-closer-444-thread-5) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:37305_solr
   [junit4]   2> 388929 INFO  (jetty-closer-444-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@42a17f46{/solr,null,UNAVAILABLE}
   [junit4]   2> 388929 INFO  (jetty-closer-444-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@499ab177{/solr,null,UNAVAILABLE}
   [junit4]   2> 388930 INFO  (jetty-closer-444-thread-2) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 388930 INFO  (jetty-closer-444-thread-4) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 388930 INFO  (zkCallback-457-thread-2) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41771_solr
   [junit4]   2> 388930 INFO  (jetty-closer-444-thread-5) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 388930 INFO  (zkCallback-457-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 388930 INFO  (zkCallback-530-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 388930 INFO  (zkCallback-537-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 388933 INFO  (jetty-closer-444-thread-5) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6109cd6{/solr,null,UNAVAILABLE}
   [junit4]   2> 388933 INFO  (jetty-closer-444-thread-5) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 390422 WARN  (zkCallback-490-thread-1) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 390424 INFO  (jetty-closer-444-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@48788af0{/solr,null,UNAVAILABLE}
   [junit4]   2> 390424 INFO  (jetty-closer-444-thread-1) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 390428 WARN  (zkCallback-473-thread-1) [    ] o.a.s.c.ZkController Unable to register nodeLost path for 127.0.0.1:37281_solr
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/nodeLost/127.0.0.1:37281_solr
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.11.jar:3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0]
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.11.jar:3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0]
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:786) ~[zookeeper-3.4.11.jar:3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0]
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$create$7(SolrZkClient.java:398) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:398) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.lambda$registerLiveNodesListener$3(ZkController.java:945) [java/:?]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.lambda$refreshLiveNodes$3(ZkStateReader.java:798) [java/:?]
   [junit4]   2> 	at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4698) [?:?]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.refreshLiveNodes(ZkStateReader.java:797) [java/:?]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.access$1400(ZkStateReader.java:73) [java/:?]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader$LiveNodeWatcher.refreshAndWatch(ZkStateReader.java:1303) [java/:?]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader$LiveNodeWatcher.process(ZkStateReader.java:1298) [java/:?]
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:270) [java/:?]
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) [java/:?]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:835) [?:?]
   [junit4]   2> 390429 INFO  (jetty-closer-444-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@15ab0b8d{/solr,null,UNAVAILABLE}
   [junit4]   2> 390429 INFO  (jetty-closer-444-thread-3) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 390430 ERROR (SUITE-TestUtilizeNode-seed#[C6E92221CB29E4BC]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 390430 INFO  (SUITE-TestUtilizeNode-seed#[C6E92221CB29E4BC]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43609 43609
   [junit4]   2> 390460 INFO  (Thread-513) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43609 43609
   [junit4]   2> 390461 WARN  (Thread-513) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	12	/solr/aliases.json
   [junit4]   2> 	8	/solr/collections/utilizenodecoll/terms/shard1
   [junit4]   2> 	5	/solr/security.json
   [junit4]   2> 	3	/solr/configs/conf1
   [junit4]   2> 	3	/solr/collections/utilizenodecoll/terms/shard2
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	12	/solr/clusterstate.json
   [junit4]   2> 	12	/solr/clusterprops.json
   [junit4]   2> 	3	/solr/collections/utilizenodecoll/state.json
   [junit4]   2> 	2	/solr/overseer_elect/election/72167881827155973-127.0.0.1:41771_solr-n_0000000002
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	14	/solr/live_nodes
   [junit4]   2> 	14	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestUtilizeNode_C6E92221CB29E4BC-003
   [junit4]   2> Nov 15, 2018 7:04:10 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1901, maxMBSortInHeap=6.42006565941725, sim=RandomSimilarity(queryNorm=false): {}, locale=mr, timezone=BET
   [junit4]   2> NOTE: Linux 4.15.0-38-generic amd64/Oracle Corporation 12-ea (64-bit)/cpus=8,threads=1,free=239705432,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestUtilizeNode, TestUtilizeNode, DeleteReplicaTest, TestUtilizeNode, TestUtilizeNode]
   [junit4] Completed [10/10 (7!)] on J2 in 45.97s, 1 test, 1 failure <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/temp/junit4-J2-20181115_185740_1271876756518659889162.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 17 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/common-build.xml:1567: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/common-build.xml:1092: There were test failures: 10 suites, 40 tests, 1 error, 7 failures [seed: C6E92221CB29E4BC]

Total time: 6 minutes 33 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   3/5 failed: org.apache.solr.cloud.DeleteReplicaTest
[repro]   4/5 failed: org.apache.solr.cloud.TestUtilizeNode
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=efae53e5e910aa1c44b201f861db856db492837d, workspace=/var/lib/jenkins/workspace/Lucene-Solr-7.x-Linux
[WARNINGS] Computing warning deltas based on reference build #3097
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2

[JENKINS] Lucene-Solr-7.x-Linux (32bit/jdk1.8.0_172) - Build # 3097 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3097/
Java: 32bit/jdk1.8.0_172 -server -XX:+UseParallelGC

2 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testParallelUpdateQTime

Error Message:
Error from server at https://127.0.0.1:43153/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason: <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>  </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:43153/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

	at __randomizedtesting.SeedInfo.seed([50646C64F8D5FADB:BEBC17F5B6AA2F48]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testParallelUpdateQTime(CloudSolrClientTest.java:146)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:43153/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more


FAILED:  org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple2 null Live Nodes: [127.0.0.1:34293_solr, 127.0.0.1:46283_solr] Last available state: DocCollection(testSimple2//collections/testSimple2/state.json/22)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "dataDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node3/data/",           "base_url":"https://127.0.0.1:46283/solr",           "node_name":"127.0.0.1:46283_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node3/data/tlog",           "core":"testSimple2_shard1_replica_n1",           "shared_storage":"true",           "state":"active",           "leader":"true"},         "core_node5":{           "dataDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node5/data/",           "base_url":"https://127.0.0.1:46759/solr",           "node_name":"127.0.0.1:46759_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node5/data/tlog",           "core":"testSimple2_shard1_replica_n2",           "shared_storage":"true",           "state":"down"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node7":{           "dataDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node7/data/",           "base_url":"https://127.0.0.1:46283/solr",           "node_name":"127.0.0.1:46283_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node7/data/tlog",           "core":"testSimple2_shard2_replica_n4",           "shared_storage":"true",           "state":"active",           "leader":"true"},         "core_node8":{           "dataDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node8/data/",           "base_url":"https://127.0.0.1:46759/solr",           "node_name":"127.0.0.1:46759_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node8/data/tlog",           "core":"testSimple2_shard2_replica_n6",           "shared_storage":"true",           "state":"down"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple2
null
Live Nodes: [127.0.0.1:34293_solr, 127.0.0.1:46283_solr]
Last available state: DocCollection(testSimple2//collections/testSimple2/state.json/22)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node3/data/",
          "base_url":"https://127.0.0.1:46283/solr",
          "node_name":"127.0.0.1:46283_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node3/data/tlog",
          "core":"testSimple2_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node5":{
          "dataDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node5/data/",
          "base_url":"https://127.0.0.1:46759/solr",
          "node_name":"127.0.0.1:46759_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node5/data/tlog",
          "core":"testSimple2_shard1_replica_n2",
          "shared_storage":"true",
          "state":"down"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node7/data/",
          "base_url":"https://127.0.0.1:46283/solr",
          "node_name":"127.0.0.1:46283_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node7/data/tlog",
          "core":"testSimple2_shard2_replica_n4",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node8":{
          "dataDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node8/data/",
          "base_url":"https://127.0.0.1:46759/solr",
          "node_name":"127.0.0.1:46759_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://localhost.localdomain:33145/data/testSimple2/core_node8/data/tlog",
          "core":"testSimple2_shard2_replica_n6",
          "shared_storage":"true",
          "state":"down"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
	at __randomizedtesting.SeedInfo.seed([5C97E18533F30E8E:6424C57B1400DA5F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
	at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:138)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14540 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_5C97E18533F30E8E-001/init-core-data-001
   [junit4]   2> 2206386 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=23 numCloses=23
   [junit4]   2> 2206386 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2206387 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2206388 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2206388 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_5C97E18533F30E8E-001/tempDir-001
   [junit4]   2> 2206388 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2206388 INFO  (Thread-4056) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2206388 INFO  (Thread-4056) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2206390 ERROR (Thread-4056) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2206488 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:43217
   [junit4]   2> 2206490 INFO  (zkConnectionManagerCallback-4758-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2206495 INFO  (jetty-launcher-4755-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2206495 INFO  (jetty-launcher-4755-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2206496 INFO  (jetty-launcher-4755-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2206496 INFO  (jetty-launcher-4755-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2206496 INFO  (jetty-launcher-4755-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2206496 INFO  (jetty-launcher-4755-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2206496 INFO  (jetty-launcher-4755-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2206496 INFO  (jetty-launcher-4755-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2206496 INFO  (jetty-launcher-4755-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@59bd07{/solr,null,AVAILABLE}
   [junit4]   2> 2206496 INFO  (jetty-launcher-4755-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@54b72f{/solr,null,AVAILABLE}
   [junit4]   2> 2206496 INFO  (jetty-launcher-4755-thread-3) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2206496 INFO  (jetty-launcher-4755-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@cee4f6{SSL,[ssl, http/1.1]}{127.0.0.1:34293}
   [junit4]   2> 2206496 INFO  (jetty-launcher-4755-thread-2) [    ] o.e.j.s.Server Started @2206527ms
   [junit4]   2> 2206497 INFO  (jetty-launcher-4755-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34293}
   [junit4]   2> 2206497 ERROR (jetty-launcher-4755-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2206497 INFO  (jetty-launcher-4755-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2206497 INFO  (jetty-launcher-4755-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 2206497 INFO  (jetty-launcher-4755-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2206497 INFO  (jetty-launcher-4755-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2206497 INFO  (jetty-launcher-4755-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-15T14:09:47.202Z
   [junit4]   2> 2206498 INFO  (jetty-launcher-4755-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@622d66{SSL,[ssl, http/1.1]}{127.0.0.1:46283}
   [junit4]   2> 2206498 INFO  (jetty-launcher-4755-thread-1) [    ] o.e.j.s.Server Started @2206529ms
   [junit4]   2> 2206498 INFO  (jetty-launcher-4755-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46283}
   [junit4]   2> 2206498 ERROR (jetty-launcher-4755-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2206498 INFO  (jetty-launcher-4755-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2206498 INFO  (jetty-launcher-4755-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 2206498 INFO  (jetty-launcher-4755-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2206498 INFO  (jetty-launcher-4755-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2206498 INFO  (jetty-launcher-4755-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-15T14:09:47.203Z
   [junit4]   2> 2206499 INFO  (jetty-launcher-4755-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2206499 INFO  (jetty-launcher-4755-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2206499 INFO  (zkConnectionManagerCallback-4762-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2206499 INFO  (jetty-launcher-4755-thread-3) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2206500 INFO  (zkConnectionManagerCallback-4760-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2206500 INFO  (jetty-launcher-4755-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1786fa5{/solr,null,AVAILABLE}
   [junit4]   2> 2206500 INFO  (jetty-launcher-4755-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2206500 INFO  (jetty-launcher-4755-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2206500 INFO  (jetty-launcher-4755-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1854ee6{SSL,[ssl, http/1.1]}{127.0.0.1:46759}
   [junit4]   2> 2206500 INFO  (jetty-launcher-4755-thread-3) [    ] o.e.j.s.Server Started @2206531ms
   [junit4]   2> 2206500 INFO  (jetty-launcher-4755-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46759}
   [junit4]   2> 2206500 ERROR (jetty-launcher-4755-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2206500 INFO  (jetty-launcher-4755-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2206500 INFO  (jetty-launcher-4755-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 2206500 INFO  (jetty-launcher-4755-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2206500 INFO  (jetty-launcher-4755-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2206500 INFO  (jetty-launcher-4755-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-15T14:09:47.205Z
   [junit4]   2> 2206501 INFO  (zkConnectionManagerCallback-4764-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2206502 INFO  (jetty-launcher-4755-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2206503 INFO  (jetty-launcher-4755-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2206503 INFO  (jetty-launcher-4755-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2206503 INFO  (jetty-launcher-4755-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2206503 INFO  (jetty-launcher-4755-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2206504 INFO  (jetty-launcher-4755-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2206504 INFO  (jetty-launcher-4755-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2206504 INFO  (jetty-launcher-4755-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1e19895, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2206504 INFO  (jetty-launcher-4755-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1e19895, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2206505 INFO  (jetty-launcher-4755-thread-3) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1e19895, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2207543 INFO  (jetty-launcher-4755-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43217/solr
   [junit4]   2> 2207544 INFO  (zkConnectionManagerCallback-4768-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2207544 DEBUG (jetty-launcher-4755-thread-2) [n:127.0.0.1:34293_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$302/7710658@1f96162
   [junit4]   2> 2207545 INFO  (zkConnectionManagerCallback-4770-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2207579 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
   [junit4]   2> 2207579 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2207580 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 2207580 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 2207580 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 2207580 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/security.json
   [junit4]   2> 2207581 DEBUG (jetty-launcher-4755-thread-2) [n:127.0.0.1:34293_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/72166727169998853-127.0.0.1:34293_solr-n_0000000000
   [junit4]   2> 2207581 INFO  (jetty-launcher-4755-thread-2) [n:127.0.0.1:34293_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34293_solr
   [junit4]   2> 2207581 INFO  (jetty-launcher-4755-thread-2) [n:127.0.0.1:34293_solr    ] o.a.s.c.Overseer Overseer (id=72166727169998853-127.0.0.1:34293_solr-n_0000000000) starting
   [junit4]   2> 2207587 INFO  (zkConnectionManagerCallback-4777-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2207588 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
   [junit4]   2> 2207588 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2207588 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 2207588 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 2207588 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 2207588 INFO  (jetty-launcher-4755-thread-2) [n:127.0.0.1:34293_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43217/solr ready
   [junit4]   2> 2207589 INFO  (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:34293_solr
   [junit4]   2> 2207590 DEBUG (OverseerCollectionConfigSetProcessor-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.OverseerTaskQueue Peeking for top 100 elements. ExcludeSet: []
   [junit4]   2> 2207590 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 2207590 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue-work
   [junit4]   2> 2207591 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 2207591 DEBUG (OverseerAutoScalingTriggerThread-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 2207591 INFO  (jetty-launcher-4755-thread-2) [n:127.0.0.1:34293_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34293_solr
   [junit4]   2> 2207592 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 2207592 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 2207592 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/autoscaling.json
   [junit4]   2> 2207592 DEBUG (OverseerAutoScalingTriggerThread-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 2207592 DEBUG (OverseerAutoScalingTriggerThread-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 2207592 DEBUG (OverseerAutoScalingTriggerThread-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 2207597 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2207597 INFO  (zkCallback-4769-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2207599 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/autoscaling/events/.scheduled_maintenance
   [junit4]   2> 2207599 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2207599 INFO  (zkCallback-4776-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2207600 DEBUG (OverseerAutoScalingTriggerThread-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:34293_solr]
   [junit4]   2> 2207603 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 2207603 DEBUG (ScheduledTrigger-7042-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 2207607 DEBUG (OverseerAutoScalingTriggerThread-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2207607 DEBUG (OverseerAutoScalingTriggerThread-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 2207611 INFO  (jetty-launcher-4755-thread-2) [n:127.0.0.1:34293_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2207625 INFO  (jetty-launcher-4755-thread-2) [n:127.0.0.1:34293_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e19895
   [junit4]   2> 2207631 INFO  (jetty-launcher-4755-thread-2) [n:127.0.0.1:34293_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e19895
   [junit4]   2> 2207631 INFO  (jetty-launcher-4755-thread-2) [n:127.0.0.1:34293_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e19895
   [junit4]   2> 2207632 INFO  (jetty-launcher-4755-thread-2) [n:127.0.0.1:34293_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_5C97E18533F30E8E-001/tempDir-001/node2/.
   [junit4]   2> 2207691 INFO  (jetty-launcher-4755-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43217/solr
   [junit4]   2> 2207691 INFO  (zkConnectionManagerCallback-4782-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2207692 DEBUG (jetty-launcher-4755-thread-3) [n:127.0.0.1:46759_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$302/7710658@56b4d
   [junit4]   2> 2207693 INFO  (zkConnectionManagerCallback-4784-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2207695 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
   [junit4]   2> 2207695 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2207695 INFO  (jetty-launcher-4755-thread-3) [n:127.0.0.1:46759_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2207695 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 2207695 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 2207696 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 2207696 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/security.json
   [junit4]   2> 2207696 DEBUG (jetty-launcher-4755-thread-3) [n:127.0.0.1:46759_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/72166727169998856-127.0.0.1:46759_solr-n_0000000001
   [junit4]   2> 2207697 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/overseer_elect/election/72166727169998853-127.0.0.1:34293_solr-n_0000000000
   [junit4]   2> 2207697 DEBUG (jetty-launcher-4755-thread-3) [n:127.0.0.1:46759_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/72166727169998853-127.0.0.1:34293_solr-n_0000000000 to know if I could be the leader
   [junit4]   2> 2207697 DEBUG (jetty-launcher-4755-thread-3) [n:127.0.0.1:46759_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:46759_solr as DOWN
   [junit4]   2> 2207697 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 2207697 INFO  (jetty-launcher-4755-thread-3) [n:127.0.0.1:46759_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2207697 INFO  (jetty-launcher-4755-thread-3) [n:127.0.0.1:46759_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46759_solr
   [junit4]   2> 2207697 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 2207697 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:46759_solr"} current state version: 0
   [junit4]   2> 2207698 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:46759_solr
   [junit4]   2> 2207698 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 2207698 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 2207698 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 2207698 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2207698 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2207698 INFO  (zkCallback-4769-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2207698 INFO  (zkCallback-4776-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2207698 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2207698 INFO  (zkCallback-4783-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2207712 INFO  (zkConnectionManagerCallback-4791-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2207713 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
   [junit4]   2> 2207713 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2207713 INFO  (jetty-launcher-4755-thread-3) [n:127.0.0.1:46759_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2207713 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 2207714 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 2207714 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 2207714 INFO  (jetty-launcher-4755-thread-3) [n:127.0.0.1:46759_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43217/solr ready
   [junit4]   2> 2207714 INFO  (jetty-launcher-4755-thread-3) [n:127.0.0.1:46759_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2207724 INFO  (jetty-launcher-4755-thread-3) [n:127.0.0.1:46759_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e19895
   [junit4]   2> 2207738 INFO  (jetty-launcher-4755-thread-3) [n:127.0.0.1:46759_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e19895
   [junit4]   2> 2207738 INFO  (jetty-launcher-4755-thread-3) [n:127.0.0.1:46759_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e19895
   [junit4]   2> 2207739 INFO  (jetty-launcher-4755-thread-3) [n:127.0.0.1:46759_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_5C97E18533F30E8E-001/tempDir-001/node3/.
   [junit4]   2> 2207798 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 2207799 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 2208236 INFO  (jetty-launcher-4755-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43217/solr
   [junit4]   2> 2208238 INFO  (zkConnectionManagerCallback-4796-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2208239 DEBUG (jetty-launcher-4755-thread-1) [n:127.0.0.1:46283_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$302/7710658@f173f2
   [junit4]   2> 2208239 INFO  (zkConnectionManagerCallback-4798-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2208242 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
   [junit4]   2> 2208242 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2208242 INFO  (jetty-launcher-4755-thread-1) [n:127.0.0.1:46283_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2208242 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 2208243 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 2208243 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 2208243 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/security.json
   [junit4]   2> 2208243 DEBUG (jetty-launcher-4755-thread-1) [n:127.0.0.1:46283_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/72166727169998859-127.0.0.1:46283_solr-n_0000000002
   [junit4]   2> 2208244 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/overseer_elect/election/72166727169998856-127.0.0.1:46759_solr-n_0000000001
   [junit4]   2> 2208244 DEBUG (jetty-launcher-4755-thread-1) [n:127.0.0.1:46283_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/72166727169998856-127.0.0.1:46759_solr-n_0000000001 to know if I could be the leader
   [junit4]   2> 2208244 DEBUG (jetty-launcher-4755-thread-1) [n:127.0.0.1:46283_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:46283_solr as DOWN
   [junit4]   2> 2208244 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 2208244 INFO  (jetty-launcher-4755-thread-1) [n:127.0.0.1:46283_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2208244 INFO  (jetty-launcher-4755-thread-1) [n:127.0.0.1:46283_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46283_solr
   [junit4]   2> 2208244 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 2208245 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:46283_solr"} current state version: 0
   [junit4]   2> 2208245 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:46283_solr
   [junit4]   2> 2208245 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 2208245 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 2208245 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 2208245 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 2208245 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
   [junit4]   2> 2208245 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2208245 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2208245 INFO  (zkCallback-4769-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2208245 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2208245 INFO  (zkCallback-4776-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2208245 INFO  (zkCallback-4783-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2208245 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2208245 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2208245 INFO  (zkCallback-4790-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2208245 INFO  (zkCallback-4797-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2208256 INFO  (zkConnectionManagerCallback-4805-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2208256 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
   [junit4]   2> 2208257 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2208257 INFO  (jetty-launcher-4755-thread-1) [n:127.0.0.1:46283_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2208257 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 2208257 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 2208257 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 2208257 INFO  (jetty-launcher-4755-thread-1) [n:127.0.0.1:46283_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43217/solr ready
   [junit4]   2> 2208257 INFO  (jetty-launcher-4755-thread-1) [n:127.0.0.1:46283_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2208274 INFO  (jetty-launcher-4755-thread-1) [n:127.0.0.1:46283_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e19895
   [junit4]   2> 2208284 INFO  (jetty-launcher-4755-thread-1) [n:127.0.0.1:46283_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e19895
   [junit4]   2> 2208284 INFO  (jetty-launcher-4755-thread-1) [n:127.0.0.1:46283_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e19895
   [junit4]   2> 2208285 INFO  (jetty-launcher-4755-thread-1) [n:127.0.0.1:46283_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_5C97E18533F30E8E-001/tempDir-001/node1/.
   [junit4]   2> 2208308 INFO  (zkConnectionManagerCallback-4808-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2208311 INFO  (zkConnectionManagerCallback-4813-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2208312 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
   [junit4]   2> 2208312 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
   [junit4]   2> 2208312 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2208312 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
   [junit4]   2> 2208312 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 2208313 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
   [junit4]   2> 2208313 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43217/solr ready
   [junit4]   2> 2208341 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr/clusterprops.json
   [junit4]   2> 2208341 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr/clusterprops.json
   [junit4]   2> 2208341 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr/clusterprops.json
   [junit4]   2> 2208341 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr/clusterprops.json
   [junit4]   2> 2208341 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr/clusterprops.json
   [junit4]   2> 2208342 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr/clusterprops.json
   [junit4]   2> 2208342 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: data: /solr/clusterprops.json
   [junit4]   2> 2208342 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
   [junit4]   2> 2208342 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
   [junit4]   2> 2208342 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
   [junit4]   2> 2208342 INFO  (qtp430107-17773) [n:127.0.0.1:46759_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster params={wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 2208342 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
   [junit4]   2> 2208342 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
   [junit4]   2> 2208342 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
   [junit4]   2> 2208342 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
   [junit4]   2> 2208345 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 2208345 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 2208486 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 2208490 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2208492 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2208505 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_localdomain_43485_hdfs____.sogwr8/webapp
   [junit4]   2> 2208605 DEBUG (ScheduledTrigger-7042-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 2208941 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:43485
   [junit4]   2> 2209146 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2209147 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2209161 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_37543_datanode____8bc3g7/webapp
   [junit4]   2> 2209591 DEBUG (OverseerCollectionConfigSetProcessor-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.OverseerTaskQueue Peeking for top 100 elements. ExcludeSet: []
   [junit4]   2> 2209605 DEBUG (ScheduledTrigger-7042-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 2209634 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37543
   [junit4]   2> 2209673 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2209674 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2209684 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_34821_datanode____.76z7oj/webapp
   [junit4]   2> 2209766 ERROR (DataNode: [[[DISK]file:/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_5C97E18533F30E8E-001/tempDir-002/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_5C97E18533F30E8E-001/tempDir-002/hdfsBaseDir/data/data2/]]  heartbeating to localhost.localdomain/127.0.0.1:33145) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 2209778 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x5ea878f590f6d: from storage DS-21a85202-6ed1-43d2-894a-76c1efd581a1 node DatanodeRegistration(127.0.0.1:39183, datanodeUuid=2589e670-933f-4a56-8eee-f73a161d9d35, infoPort=40295, infoSecurePort=0, ipcPort=40933, storageInfo=lv=-56;cid=testClusterID;nsid=100083511;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 2209778 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x5ea878f590f6d: from storage DS-7101b99f-de9a-4b83-82a6-3974df962e4f node DatanodeRegistration(127.0.0.1:39183, datanodeUuid=2589e670-933f-4a56-8eee-f73a161d9d35, infoPort=40295, infoSecurePort=0, ipcPort=40933, storageInfo=lv=-56;cid=testClusterID;nsid=100083511;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 2210139 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[5C97E18533F30E8E]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34821
   [junit4]   2> 2210593 ERROR (DataNode: [[[DISK]file:/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_5C97E18533F30E8E-001/tempDir-002/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_5C97E18533F30E8E-001/tempDir-002/hdfsBaseDir/data/data4/]]  heartbeating to localhost.localdomain/127.0.0.1:33145) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 2210597 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x5ea87c030b198: from storage DS-11c4b1ea-f17f-48db-8f8b-de0fe89b63fa node DatanodeRegistration(127.0.0.1:37749, datanodeUuid=0032fdfa-14de-47ed-845f-71685323c54a, infoPort=33435, infoSecurePort=0, ipcPort=37219, storageInfo=lv=-56;cid=testClusterID;nsid=100083511;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 2210597 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x5ea87c030b198: from storage DS-d4abbc4c-dde2-4edf-aba0-b1a34d62e01a node DatanodeRegistration(127.0.0.1:37749, datanodeUuid=0032fdfa-14de-47ed-845f-71685323c54a, infoPort=33435, infoSecurePort=0, ipcPort=37219, storageInfo=lv=-56;cid=testClusterID;nsid=100083511;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 2210605 DEBUG (ScheduledTrigger-7042-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 2210747 INFO  (TEST-HdfsAutoAddReplicasIntegrationTest.testSimple-seed#[5C97E18533F30E8E]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSimple
   [junit4]   2> 2210747 INFO  (qtp430107-17774) [n:127.0.0.1:46759_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf&maxShardsPerNode=2&autoAddReplicas=true&name=testSimple1&nrtReplicas=2&action=CREATE&numShards=2&createNodeSet=127.0.0.1:34293_solr,127.0.0.1:46759_solr&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2210748 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 2210748 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/overseer/collection-queue-work/qnr-0000000000
   [junit4]   2> 2210748 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 2210748 DEBUG (OverseerCollectionConfigSetProcessor-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.ZkDistributedQueue Found child node with improper name: qnr-0000000000
   [junit4]   2> 2210748 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
   [junit4]   2> 2210749 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
   [junit4]   2> 2210749 DEBUG (OverseerCollectionConfigSetProcessor-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.ZkDistributedQueue Found child node with improper name: qnr-0000000000
   [junit4]   2> 2210749 DEBUG (OverseerCollectionConfigSetProcessor-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.OverseerTaskQueue Returning topN elements: [/overseer/collection-queue-work/qn-0000000000, ]
   [junit4]   2> 2210749 INFO  (OverseerThreadFactory-7044-thread-1-processing-n:127.0.0.1:34293_solr) [n:127.0.0.1:34293_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testSimple1
   [junit4]   2> 2210749 DEBUG (OverseerCollectionConfigSetProcessor-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.OverseerTaskQueue Peeking for top 99 elements. ExcludeSet: [/overseer/collection-queue-work/qn-0000000000],[]
   [junit4]   2> 2210750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 2210750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 2210750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 2210750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 2210750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 2210750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 2210750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/collections
   [junit4]   2> 2210750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 2210750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 2210750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 2210750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 2210750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 2210750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 2210750 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
   [junit4]   2> 2210751 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 2210751 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 2210751 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"testSimple1",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"conf",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "maxShardsPerNode":"2",
   [junit4]   2>   "createNodeSet":"127.0.0.1:34293_solr,127.0.0.1:46759_solr",
   [junit4]   2>   "autoAddReplicas":"true",
   [junit4]   2>   "nrtReplicas":"2",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "replicationFactor":"2",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 2210751 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.ClusterStateMutator building a new cName: testSimple1
   [junit4]   2> 2210751 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/testSimple1/state.json
   [junit4]   2> 2210751 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 2210752 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 2210854 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 2210855 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 2210855 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testSimple1_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 2210855 INFO  (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testSimple1_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2210855 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.SliceMutator Old Slice: shard1:{
   [junit4]   2>   "range":"80000000-ffffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{}}
   [junit4]   2> 2210855 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.SliceMutator New Slice: shard1:{
   [junit4]   2>   "range":"80000000-ffffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{"core_node3":{
   [junit4]   2>       "core":"testSimple1_shard1_replica_n1",
   [junit4]   2>       "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 2210855 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 2210856 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 2210856 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testSimple1_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:46759/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 2210856 INFO  (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testSimple1_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:46759/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2210856 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.SliceMutator Old Slice: shard1:{
   [junit4]   2>   "range":"80000000-ffffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{"core_node3":{
   [junit4]   2>       "core":"testSimple1_shard1_replica_n1",
   [junit4]   2>       "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 2210856 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.SliceMutator New Slice: shard1:{
   [junit4]   2>   "range":"80000000-ffffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{
   [junit4]   2>     "core_node3":{
   [junit4]   2>       "core":"testSimple1_shard1_replica_n1",
   [junit4]   2>       "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"},
   [junit4]   2>     "core_node5":{
   [junit4]   2>       "core":"testSimple1_shard1_replica_n2",
   [junit4]   2>       "base_url":"https://127.0.0.1:46759/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 2210857 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 2210857 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 2210857 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testSimple1_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 2210857 INFO  (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testSimple1_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2210857 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.SliceMutator Old Slice: shard2:{
   [junit4]   2>   "range":"0-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{}}
   [junit4]   2> 2210857 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.SliceMutator New Slice: shard2:{
   [junit4]   2>   "range":"0-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{"core_node7":{
   [junit4]   2>       "core":"testSimple1_shard2_replica_n4",
   [junit4]   2>       "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 2210858 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 2210858 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 2210858 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.Overseer processMessage: queueSize: 4, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testSimple1_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:46759/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 2210858 INFO  (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testSimple1_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:46759/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2210858 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.SliceMutator Old Slice: shard2:{
   [junit4]   2>   "range":"0-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{"core_node7":{
   [junit4]   2>       "core":"testSimple1_shard2_replica_n4",
   [junit4]   2>       "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 2210858 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.SliceMutator New Slice: shard2:{
   [junit4]   2>   "range":"0-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{
   [junit4]   2>     "core_node7":{
   [junit4]   2>       "core":"testSimple1_shard2_replica_n4",
   [junit4]   2>       "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"},
   [junit4]   2>     "core_node8":{
   [junit4]   2>       "core":"testSimple1_shard2_replica_n6",
   [junit4]   2>       "base_url":"https://127.0.0.1:46759/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 2210959 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/testSimple1/state.json version: 0
   [junit4]   2> 2210959 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 2210959 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 2211072 INFO  (qtp430107-17775) [n:127.0.0.1:46759_solr    x:testSimple1_shard2_replica_n6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n6&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2211072 INFO  (qtp430107-17766) [n:127.0.0.1:46759_solr    x:testSimple1_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n2&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2211074 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/collections/testSimple1/state.json
   [junit4]   2> 2211074 DEBUG (qtp430107-17775) [n:127.0.0.1:46759_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 2211074 DEBUG (qtp430107-17766) [n:127.0.0.1:46759_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 2211074 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 2211075 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 2211075 DEBUG (qtp430107-17775) [n:127.0.0.1:46759_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.ZkController Registering watch for collection testSimple1
   [junit4]   2> 2211075 DEBUG (qtp430107-17775) [n:127.0.0.1:46759_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.ZkController waiting to find shard id in clusterstate for testSimple1_shard2_replica_n6
   [junit4]   2> 2211075 DEBUG (qtp430107-17766) [n:127.0.0.1:46759_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.ZkController Registering watch for collection testSimple1
   [junit4]   2> 2211075 DEBUG (qtp430107-17766) [n:127.0.0.1:46759_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.ZkController waiting to find shard id in clusterstate for testSimple1_shard1_replica_n2
   [junit4]   2> 2211075 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "core":"testSimple1_shard2_replica_n6",
   [junit4]   2>   "core_node_name":"core_node8",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:46759/solr",
   [junit4]   2>   "node_name":"127.0.0.1:46759_solr",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "force_set_state":"false",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2211075 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"testSimple1_shard2_replica_n6",
   [junit4]   2>   "core_node_name":"core_node8",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:46759/solr",
   [junit4]   2>   "node_name":"127.0.0.1:46759_solr",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "force_set_state":"false",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 2211075 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.ReplicaMutator Will update state for replica: core_node8:{"core":"testSimple1_shard2_replica_n6","base_url":"https://127.0.0.1:46759/solr","node_name":"127.0.0.1:46759_solr","state":"down","type":"NRT","force_set_state":"false"}
   [junit4]   2> 2211075 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.ReplicaMutator Collection is now: DocCollection(testSimple1//collections/testSimple1/state.json/1)={
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "replicationFactor":"2",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"2",
   [junit4]   2>   "autoAddReplicas":"true",
   [junit4]   2>   "nrtReplicas":"2",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "shards":{
   [junit4]   2>     "shard1":{
   [junit4]   2>       "range":"80000000-ffffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node3":{
   [junit4]   2>           "core":"testSimple1_shard1_replica_n1",
   [junit4]   2>           "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node5":{
   [junit4]   2>           "core":"testSimple1_shard1_replica_n2",
   [junit4]   2>           "base_url":"https://127.0.0.1:46759/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"}}},
   [junit4]   2>     "shard2":{
   [junit4]   2>       "range":"0-7fffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node7":{
   [junit4]   2>           "core":"testSimple1_shard2_replica_n4",
   [junit4]   2>           "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node8":{
   [junit4]   2>           "core":"testSimple1_shard2_replica_n6",
   [junit4]   2>           "base_url":"https://127.0.0.1:46759/solr",
   [junit4]   2>           "node_name":"127.0.0.1:46759_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "force_set_state":"false"}}}}}
   [junit4]   2> 2211076 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "core":"testSimple1_shard1_replica_n2",
   [junit4]   2>   "core_node_name":"core_node5",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:46759/solr",
   [junit4]   2>   "node_name":"127.0.0.1:46759_solr",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "force_set_state":"false",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2211076 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"testSimple1_shard1_replica_n2",
   [junit4]   2>   "core_node_name":"core_node5",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:46759/solr",
   [junit4]   2>   "node_name":"127.0.0.1:46759_solr",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "force_set_state":"false",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 2211076 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.ReplicaMutator Will update state for replica: core_node5:{"core":"testSimple1_shard1_replica_n2","base_url":"https://127.0.0.1:46759/solr","node_name":"127.0.0.1:46759_solr","state":"down","type":"NRT","force_set_state":"false"}
   [junit4]   2> 2211076 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.ReplicaMutator Collection is now: DocCollection(testSimple1//collections/testSimple1/state.json/1)={
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "replicationFactor":"2",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"2",
   [junit4]   2>   "autoAddReplicas":"true",
   [junit4]   2>   "nrtReplicas":"2",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "shards":{
   [junit4]   2>     "shard1":{
   [junit4]   2>       "range":"80000000-ffffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node3":{
   [junit4]   2>           "core":"testSimple1_shard1_replica_n1",
   [junit4]   2>           "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node5":{
   [junit4]   2>           "core":"testSimple1_shard1_replica_n2",
   [junit4]   2>           "base_url":"https://127.0.0.1:46759/solr",
   [junit4]   2>           "node_name":"127.0.0.1:46759_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "force_set_state":"false"}}},
   [junit4]   2>     "shard2":{
   [junit4]   2>       "range":"0-7fffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node7":{
   [junit4]   2>           "core":"testSimple1_shard2_replica_n4",
   [junit4]   2>           "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node8":{
   [junit4]   2>           "core":"testSimple1_shard2_replica_n6",
   [junit4]   2>           "base_url":"https://127.0.0.1:46759/solr",
   [junit4]   2>           "node_name":"127.0.0.1:46759_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT",
   [junit4]   2>           "force_set_state":"false"}}}}}
   [junit4]   2> 2211092 INFO  (qtp954889-17757) [n:127.0.0.1:34293_solr    x:testSimple1_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n1&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2211092 INFO  (qtp954889-17757) [n:127.0.0.1:34293_solr    x:testSimple1_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2211094 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: data: /solr/collections/testSimple1/state.json
   [junit4]   2> 2211095 DEBUG (qtp954889-17757) [n:127.0.0.1:34293_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 2211097 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
   [junit4]   2> 2211098 DEBUG (SyncThread:0) [    ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
   [junit4]   2> 2211098 DEBUG (qtp954889-17757) [n:127.0.0.1:34293_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ZkController Registering watch for collection testSimple1
   [junit4]   2> 2211098 DEBUG (qtp954889-17757) [n:127.0.0.1:34293_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ZkController waiting to find shard id in clusterstate for testSimple1_shard1_replica_n1
   [junit4]   2> 2211098 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
   [junit4]   2>   "core":"testSimple1_shard1_replica_n1",
   [junit4]   2>   "core_node_name":"core_node3",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>   "node_name":"127.0.0.1:34293_solr",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "force_set_state":"false",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2211098 DEBUG (OverseerStateUpdate-72166727169998853-127.0.0.1:34293_solr-n_0000000000) [n:127.0.0.1:34293_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"testSimple1_shard1_replica_n1",
   [junit4]   2>   "core_node_name":"core_node3",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:34293/solr",
   [junit4]   2>   "node_name":"127.0.0.1:34293_solr",

[...truncated too long message...]

ty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
   [junit4]   1> 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
   [junit4]   1> 	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
   [junit4]   1> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
   [junit4]   1> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   1> 
   [junit4]   1> 14:24:21.555 [qtp334489-1997] ERROR org.apache.solr.handler.RequestHandlerBase - org.apache.solr.common.SolrException: Collection: boguscollectionname not found
   [junit4]   1> 	at org.apache.solr.handler.admin.ClusterStatus.getClusterStatus(ClusterStatus.java:115)
   [junit4]   1> 	at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$22(CollectionsHandler.java:865)
   [junit4]   1> 	at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1177)
   [junit4]   1> 	at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:258)
   [junit4]   1> 	at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:246)
   [junit4]   1> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]   1> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)
   [junit4]   1> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)
   [junit4]   1> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
   [junit4]   1> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
   [junit4]   1> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
   [junit4]   1> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   1> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
   [junit4]   1> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
   [junit4]   1> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
   [junit4]   1> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   1> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
   [junit4]   1> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
   [junit4]   1> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   1> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
   [junit4]   1> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   1> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
   [junit4]   1> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   1> 	at org.eclipse.jetty.server.Server.handle(Server.java:531)
   [junit4]   1> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
   [junit4]   1> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   1> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
   [junit4]   1> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
   [junit4]   1> 	at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
   [junit4]   1> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> 	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   1> 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
   [junit4]   1> 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
   [junit4]   1> 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
   [junit4]   1> 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
   [junit4]   1> 	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
   [junit4]   1> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
   [junit4]   1> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   1> 
   [junit4]   1> 14:24:32.989 [SUITE-CloudSolrClientTest-seed#[50646C64F8D5FADB]-worker] ERROR org.apache.zookeeper.server.ZooKeeperServer - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.impl.CloudSolrClientTest_50646C64F8D5FADB-001
   [junit4]   2> Nov 15, 2018 2:24:33 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {multiDefault=FSTOrd50, a_t=FST50, title_s=FSTOrd50, id=Lucene50(blocksize=128), text=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{_version_=DocValuesFormat(name=Memory), multiDefault=DocValuesFormat(name=Lucene70), a_t=DocValuesFormat(name=Memory), title_s=DocValuesFormat(name=Lucene70), intDefault=DocValuesFormat(name=Memory), id=DocValuesFormat(name=Direct), text=DocValuesFormat(name=Asserting), timestamp=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=1824, maxMBSortInHeap=6.949506104808478, sim=RandomSimilarity(queryNorm=true): {}, locale=es-GT, timezone=America/Los_Angeles
   [junit4]   2> NOTE: Linux 4.15.0-38-generic i386/Oracle Corporation 1.8.0_172 (32-bit)/cpus=8,threads=1,free=47438600,total=302514176
   [junit4]   2> NOTE: All tests run in this JVM: [HttpSolrClientBuilderTest, TestCloudCollectionsListeners, CloudSolrClientMultiConstructorTest, StreamExpressionToExpessionTest, TestDocumentObjectBinder, CloudSolrClientBadInputTest, TestJsonRecordReader, AbsoluteValueEvaluatorTest, UniformDistributionEvaluatorTest, SolrDocumentTest, TestEmbeddedSolrServer, GetByIdTest, ConcurrentUpdateSolrClientBadInputTest, NormalDistributionEvaluatorTest, SelectWithEvaluatorsTest, SolrExceptionTest, DirectJsonQueryRequestFacetingIntegrationTest, CorrelationEvaluatorTest, JsonValidatorTest, TestRetryUtil, FloorEvaluatorTest, TermsResponseTest, TestBatchUpdate, ConcatOperationTest, ModuloEvaluatorTest, RecursiveEvaluatorTest, JdbcTest, CeilingEvaluatorTest, UsingSolrJRefGuideExamplesTest, ZkConfigFilesTest, JettyWebappTest, LargeVolumeBinaryJettyTest, LargeVolumeEmbeddedTest, LargeVolumeJettyTest, MergeIndexesEmbeddedTest, SolrExampleJettyTest, SolrExampleStreamingBinaryTest, SolrExampleStreamingTest, BasicHttpSolrClientTest, CloudSolrClientCacheTest, CloudSolrClientTest]
   [junit4] Completed [143/172 (1!)] on J1 in 60.34s, 16 tests, 1 error <<< FAILURES!

[...truncated 44015 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3097/consoleText

[repro] Revision: efae53e5e910aa1c44b201f861db856db492837d

[repro] Ant options: "-Dargs=-server -XX:+UseParallelGC"
[repro] Repro line:  ant test  -Dtestcase=HdfsAutoAddReplicasIntegrationTest -Dtests.method=testSimple -Dtests.seed=5C97E18533F30E8E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es-CO -Dtests.timezone=Asia/Katmandu -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=CloudSolrClientTest -Dtests.method=testParallelUpdateQTime -Dtests.seed=50646C64F8D5FADB -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es-GT -Dtests.timezone=America/Los_Angeles -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/solrj
[repro]       CloudSolrClientTest
[repro]    solr/core
[repro]       HdfsAutoAddReplicasIntegrationTest
[repro] ant compile-test

[...truncated 2717 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.CloudSolrClientTest" -Dtests.showOutput=onerror "-Dargs=-server -XX:+UseParallelGC" -Dtests.seed=50646C64F8D5FADB -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es-GT -Dtests.timezone=America/Los_Angeles -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[...truncated 146 lines...]
[repro] ant compile-test

[...truncated 1352 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.HdfsAutoAddReplicasIntegrationTest" -Dtests.showOutput=onerror "-Dargs=-server -XX:+UseParallelGC" -Dtests.seed=5C97E18533F30E8E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es-CO -Dtests.timezone=Asia/Katmandu -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[...truncated 71 lines...]
[repro] Failures:
[repro]   0/5 failed: org.apache.solr.client.solrj.impl.CloudSolrClientTest
[repro]   0/5 failed: org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest
[repro] Exiting with code 0

[...truncated 40 lines...]

[JENKINS-EA] Lucene-Solr-7.x-Linux (64bit/jdk-12-ea+12) - Build # 3096 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3096/
Java: 64bit/jdk-12-ea+12 -XX:+UseCompressedOops -XX:+UseParallelGC

23 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting

Error Message:
Error from server at http://127.0.0.1:46865/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason: <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>  </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:46865/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

	at __randomizedtesting.SeedInfo.seed([B70CE2DD156BAB96:75BBDEB5162B5BEE]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting(CloudSolrClientTest.java:238)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:46865/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	... 1 more


FAILED:  org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.testSslAndNoClientAuth

Error Message:
Could not find collection:second_collection

Stack Trace:
java.lang.AssertionError: Could not find collection:second_collection
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:92FFEABC0633892]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkCreateCollection(TestMiniSolrCloudClusterSSL.java:263)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkClusterWithCollectionCreations(TestMiniSolrCloudClusterSSL.java:249)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkClusterWithNodeReplacement(TestMiniSolrCloudClusterSSL.java:157)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.testSslAndNoClientAuth(TestMiniSolrCloudClusterSSL.java:121)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.testSslAndNoClientAuth

Error Message:
Could not find collection:second_collection

Stack Trace:
java.lang.AssertionError: Could not find collection:second_collection
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:92FFEABC0633892]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkCreateCollection(TestMiniSolrCloudClusterSSL.java:263)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkClusterWithCollectionCreations(TestMiniSolrCloudClusterSSL.java:249)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkClusterWithNodeReplacement(TestMiniSolrCloudClusterSSL.java:157)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.testSslAndNoClientAuth(TestMiniSolrCloudClusterSSL.java:121)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.testSslWithInvalidPeerName

Error Message:
Could not find collection:second_collection

Stack Trace:
java.lang.AssertionError: Could not find collection:second_collection
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:A2246F24F8BF7649]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkCreateCollection(TestMiniSolrCloudClusterSSL.java:263)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkClusterWithCollectionCreations(TestMiniSolrCloudClusterSSL.java:249)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.testSslWithInvalidPeerName(TestMiniSolrCloudClusterSSL.java:185)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.testSslAndNoClientAuth

Error Message:
Could not find collection:second_collection

Stack Trace:
java.lang.AssertionError: Could not find collection:second_collection
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:92FFEABC0633892]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkCreateCollection(TestMiniSolrCloudClusterSSL.java:263)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkClusterWithCollectionCreations(TestMiniSolrCloudClusterSSL.java:249)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkClusterWithNodeReplacement(TestMiniSolrCloudClusterSSL.java:157)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.testSslAndNoClientAuth(TestMiniSolrCloudClusterSSL.java:121)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.testSslWithInvalidPeerName

Error Message:
Could not find collection:second_collection

Stack Trace:
java.lang.AssertionError: Could not find collection:second_collection
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:A2246F24F8BF7649]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkCreateCollection(TestMiniSolrCloudClusterSSL.java:263)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkClusterWithCollectionCreations(TestMiniSolrCloudClusterSSL.java:249)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.testSslWithInvalidPeerName(TestMiniSolrCloudClusterSSL.java:185)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.testSslAndNoClientAuth

Error Message:
Could not find collection:second_collection

Stack Trace:
java.lang.AssertionError: Could not find collection:second_collection
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:92FFEABC0633892]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkCreateCollection(TestMiniSolrCloudClusterSSL.java:263)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkClusterWithCollectionCreations(TestMiniSolrCloudClusterSSL.java:249)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkClusterWithNodeReplacement(TestMiniSolrCloudClusterSSL.java:157)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.testSslAndNoClientAuth(TestMiniSolrCloudClusterSSL.java:121)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.testSslAndNoClientAuth

Error Message:
Could not find collection:second_collection

Stack Trace:
java.lang.AssertionError: Could not find collection:second_collection
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:92FFEABC0633892]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkCreateCollection(TestMiniSolrCloudClusterSSL.java:263)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkClusterWithCollectionCreations(TestMiniSolrCloudClusterSSL.java:249)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.checkClusterWithNodeReplacement(TestMiniSolrCloudClusterSSL.java:157)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterSSL.testSslAndNoClientAuth(TestMiniSolrCloudClusterSSL.java:121)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Could not find collection : testSimple1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : testSimple1
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:CD260E611FB05D89]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
	at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.getReplacedSharedFsReplicas(AutoAddReplicasIntegrationTest.java:181)
	at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:98)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Could not find collection : testSimple1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : testSimple1
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:CD260E611FB05D89]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
	at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.getReplacedSharedFsReplicas(AutoAddReplicasIntegrationTest.java:181)
	at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:98)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Could not find collection : testSimple1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : testSimple1
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:CD260E611FB05D89]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
	at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.getReplacedSharedFsReplicas(AutoAddReplicasIntegrationTest.java:181)
	at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:98)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Could not find collection : testSimple1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : testSimple1
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:CD260E611FB05D89]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
	at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.getReplacedSharedFsReplicas(AutoAddReplicasIntegrationTest.java:181)
	at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:98)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.handler.component.DistributedMLTComponentTest.test

Error Message:
Error from server at https://127.0.0.1:44171//collection1: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:33359//collection1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:44171//collection1: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:33359//collection1
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:7DC1154596BFE4A0]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:983)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:998)
	at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:559)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:611)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:589)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:568)
	at org.apache.solr.handler.component.DistributedMLTComponentTest.test(DistributedMLTComponentTest.java:153)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.handler.component.DistributedMLTComponentTest.test

Error Message:
Error from server at https://127.0.0.1:45765//collection1: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:38883//collection1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:45765//collection1: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:38883//collection1
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:7DC1154596BFE4A0]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:983)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:998)
	at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:559)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:611)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:589)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:568)
	at org.apache.solr.handler.component.DistributedMLTComponentTest.test(DistributedMLTComponentTest.java:153)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.handler.component.DistributedMLTComponentTest.test

Error Message:
Error from server at https://127.0.0.1:36337//collection1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://[ff01::114]:33332, https://[ff01::083]:33332, https://[ff01::213]:33332, https://127.0.0.1:34457//collection1]

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:36337//collection1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://[ff01::114]:33332, https://[ff01::083]:33332, https://[ff01::213]:33332, https://127.0.0.1:34457//collection1]
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:7DC1154596BFE4A0]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:983)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:998)
	at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:559)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:611)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:589)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:568)
	at org.apache.solr.handler.component.DistributedMLTComponentTest.test(DistributedMLTComponentTest.java:148)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test

Error Message:
Collection not found: backuprestore

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: backuprestore
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:7DC1154596BFE4A0]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:851)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.indexDocs(AbstractCloudBackupRestoreTestCase.java:219)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:127)
	at org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:64)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test

Error Message:
Collection not found: backuprestore

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: backuprestore
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:7DC1154596BFE4A0]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:851)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.indexDocs(AbstractCloudBackupRestoreTestCase.java:219)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:127)
	at org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:64)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test

Error Message:
Collection not found: backuprestore

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: backuprestore
	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:7DC1154596BFE4A0]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:851)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.indexDocs(AbstractCloudBackupRestoreTestCase.java:219)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:127)
	at org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:64)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapte

[...truncated too long message...]

istry solr.jetty / com.codahale.metrics.MetricRegistry@7880528
   [junit4]   2> 307627 INFO  (coreCloseExecutor-1046-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@2ef47ff7
   [junit4]   2> 307627 INFO  (coreCloseExecutor-1046-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=2ef47ff7
   [junit4]   2> 307627 INFO  (coreCloseExecutor-1046-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@580e0f14: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@37b69966
   [junit4]   2> 307634 INFO  (TEST-DistributedMLTComponentTest.test-seed#[F5952A9F38438958]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@306d4a34{/,null,UNAVAILABLE}
   [junit4]   2> 307634 INFO  (TEST-DistributedMLTComponentTest.test-seed#[F5952A9F38438958]) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 307641 INFO  (TEST-DistributedMLTComponentTest.test-seed#[F5952A9F38438958]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@43ddaf0d{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 307642 INFO  (TEST-DistributedMLTComponentTest.test-seed#[F5952A9F38438958]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=943408793
   [junit4]   2> 307642 INFO  (TEST-DistributedMLTComponentTest.test-seed#[F5952A9F38438958]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 307642 INFO  (TEST-DistributedMLTComponentTest.test-seed#[F5952A9F38438958]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@50fe7685: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@6aebcab7
   [junit4]   2> 307648 INFO  (TEST-DistributedMLTComponentTest.test-seed#[F5952A9F38438958]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 307648 INFO  (TEST-DistributedMLTComponentTest.test-seed#[F5952A9F38438958]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@29f3268c: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@1234d20f
   [junit4]   2> 307652 INFO  (TEST-DistributedMLTComponentTest.test-seed#[F5952A9F38438958]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 307652 INFO  (TEST-DistributedMLTComponentTest.test-seed#[F5952A9F38438958]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5abe3979: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@7880528
   [junit4]   2> 307653 INFO  (coreCloseExecutor-1048-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@24887697
   [junit4]   2> 307653 INFO  (coreCloseExecutor-1048-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=24887697
   [junit4]   2> 307653 INFO  (coreCloseExecutor-1048-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4676aaab: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@115d0859
   [junit4]   2> 307672 INFO  (TEST-DistributedMLTComponentTest.test-seed#[F5952A9F38438958]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5e150991{/,null,UNAVAILABLE}
   [junit4]   2> 307672 INFO  (TEST-DistributedMLTComponentTest.test-seed#[F5952A9F38438958]) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DistributedMLTComponentTest -Dtests.method=test -Dtests.seed=F5952A9F38438958 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=kln -Dtests.timezone=America/Argentina/Mendoza -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   3.86s J2 | DistributedMLTComponentTest.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:45765//collection1: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:38883//collection1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F5952A9F38438958:7DC1154596BFE4A0]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:983)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:998)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:559)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:611)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:589)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:568)
   [junit4]    > 	at org.apache.solr.handler.component.DistributedMLTComponentTest.test(DistributedMLTComponentTest.java:153)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
   [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:835)
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.handler.component.DistributedMLTComponentTest_F5952A9F38438958-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=true): {}, locale=kln, timezone=America/Argentina/Mendoza
   [junit4]   2> NOTE: Linux 4.15.0-38-generic amd64/Oracle Corporation 12-ea (64-bit)/cpus=8,threads=1,free=295863208,total=429391872
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsAutoAddReplicasIntegrationTest, HdfsAutoAddReplicasIntegrationTest, TestLocalFSCloudBackupRestore, TestMiniSolrCloudClusterSSL, HdfsAutoAddReplicasIntegrationTest, DistributedMLTComponentTest]
   [junit4] Completed [20/20 (12!)] on J2 in 3.88s, 1 test, 1 error <<< FAILURES!

[...truncated 20 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/common-build.xml:1567: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/common-build.xml:1092: There were test failures: 20 suites, 45 tests, 12 errors, 6 failures [seed: F5952A9F38438958]

Total time: 5 minutes 10 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.client.solrj.impl.CloudSolrClientTest
[repro]   2/5 failed: org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore
[repro]   2/5 failed: org.apache.solr.handler.component.DistributedMLTComponentTest
[repro]   3/5 failed: org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest
[repro]   5/5 failed: org.apache.solr.cloud.TestMiniSolrCloudClusterSSL
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=efae53e5e910aa1c44b201f861db856db492837d, workspace=/var/lib/jenkins/workspace/Lucene-Solr-7.x-Linux
[WARNINGS] Computing warning deltas based on reference build #3095
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-10.0.1) - Build # 3095 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3095/
Java: 64bit/jdk-10.0.1 -XX:+UseCompressedOops -XX:+UseParallelGC

11 tests failed.
FAILED:  org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test

Error Message:
Node 127.0.0.1:38451_solr has 3 replicas. Expected num replicas : 2. state:  DocCollection(hdfsbackuprestore_restored//collections/hdfsbackuprestore_restored/state.json/12)={   "pullReplicas":0,   "replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node62":{           "core":"hdfsbackuprestore_restored_shard2_replica_n61",           "base_url":"http://127.0.0.1:38451/solr",           "node_name":"127.0.0.1:38451_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265945388056525"},     "shard1_1":{       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{"core_node64":{           "core":"hdfsbackuprestore_restored_shard1_1_replica_n63",           "base_url":"http://127.0.0.1:38451/solr",           "node_name":"127.0.0.1:38451_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265945388093775"},     "shard1_0":{       "range":"80000000-bfffffff",       "state":"active",       "replicas":{"core_node66":{           "core":"hdfsbackuprestore_restored_shard1_0_replica_n65",           "base_url":"http://127.0.0.1:38451/solr",           "node_name":"127.0.0.1:38451_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265945388123048"}},   "router":{     "name":"compositeId",     "field":"shard_s"},   "maxShardsPerNode":"-1",   "autoAddReplicas":"true",   "nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:38451_solr has 3 replicas. Expected num replicas : 2. state: 
DocCollection(hdfsbackuprestore_restored//collections/hdfsbackuprestore_restored/state.json/12)={
  "pullReplicas":0,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node62":{
          "core":"hdfsbackuprestore_restored_shard2_replica_n61",
          "base_url":"http://127.0.0.1:38451/solr",
          "node_name":"127.0.0.1:38451_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265945388056525"},
    "shard1_1":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{"core_node64":{
          "core":"hdfsbackuprestore_restored_shard1_1_replica_n63",
          "base_url":"http://127.0.0.1:38451/solr",
          "node_name":"127.0.0.1:38451_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265945388093775"},
    "shard1_0":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{"core_node66":{
          "core":"hdfsbackuprestore_restored_shard1_0_replica_n65",
          "base_url":"http://127.0.0.1:38451/solr",
          "node_name":"127.0.0.1:38451_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265945388123048"}},
  "router":{
    "name":"compositeId",
    "field":"shard_s"},
  "maxShardsPerNode":"-1",
  "autoAddReplicas":"true",
  "nrtReplicas":1,
  "tlogReplicas":0}
	at __randomizedtesting.SeedInfo.seed([864380771B02AB2:803007DDDF4C474A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
	at java.base/java.util.HashMap.forEach(HashMap.java:1341)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
	at org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test(TestHdfsCloudBackupRestore.java:213)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test

Error Message:
Node 127.0.0.1:38105_solr has 3 replicas. Expected num replicas : 2. state:  DocCollection(hdfsbackuprestore_restored//collections/hdfsbackuprestore_restored/state.json/12)={   "pullReplicas":0,   "replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node62":{           "core":"hdfsbackuprestore_restored_shard2_replica_n61",           "base_url":"http://127.0.0.1:38105/solr",           "node_name":"127.0.0.1:38105_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265956818476689"},     "shard1_1":{       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{"core_node64":{           "core":"hdfsbackuprestore_restored_shard1_1_replica_n63",           "base_url":"http://127.0.0.1:38105/solr",           "node_name":"127.0.0.1:38105_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265956818515377"},     "shard1_0":{       "range":"80000000-bfffffff",       "state":"active",       "replicas":{"core_node66":{           "core":"hdfsbackuprestore_restored_shard1_0_replica_n65",           "base_url":"http://127.0.0.1:38105/solr",           "node_name":"127.0.0.1:38105_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265956818545589"}},   "router":{     "name":"compositeId",     "field":"shard_s"},   "maxShardsPerNode":"-1",   "autoAddReplicas":"true",   "nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:38105_solr has 3 replicas. Expected num replicas : 2. state: 
DocCollection(hdfsbackuprestore_restored//collections/hdfsbackuprestore_restored/state.json/12)={
  "pullReplicas":0,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node62":{
          "core":"hdfsbackuprestore_restored_shard2_replica_n61",
          "base_url":"http://127.0.0.1:38105/solr",
          "node_name":"127.0.0.1:38105_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265956818476689"},
    "shard1_1":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{"core_node64":{
          "core":"hdfsbackuprestore_restored_shard1_1_replica_n63",
          "base_url":"http://127.0.0.1:38105/solr",
          "node_name":"127.0.0.1:38105_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265956818515377"},
    "shard1_0":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{"core_node66":{
          "core":"hdfsbackuprestore_restored_shard1_0_replica_n65",
          "base_url":"http://127.0.0.1:38105/solr",
          "node_name":"127.0.0.1:38105_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265956818545589"}},
  "router":{
    "name":"compositeId",
    "field":"shard_s"},
  "maxShardsPerNode":"-1",
  "autoAddReplicas":"true",
  "nrtReplicas":1,
  "tlogReplicas":0}
	at __randomizedtesting.SeedInfo.seed([864380771B02AB2:803007DDDF4C474A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
	at java.base/java.util.HashMap.forEach(HashMap.java:1341)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
	at org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test(TestHdfsCloudBackupRestore.java:213)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test

Error Message:
Node 127.0.0.1:41321_solr has 3 replicas. Expected num replicas : 2. state:  DocCollection(hdfsbackuprestore_restored//collections/hdfsbackuprestore_restored/state.json/11)={   "pullReplicas":0,   "replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node62":{           "core":"hdfsbackuprestore_restored_shard2_replica_n61",           "base_url":"http://127.0.0.1:41321/solr",           "node_name":"127.0.0.1:41321_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265982814062346"},     "shard1_1":{       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{"core_node64":{           "core":"hdfsbackuprestore_restored_shard1_1_replica_n63",           "base_url":"http://127.0.0.1:41321/solr",           "node_name":"127.0.0.1:41321_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265982814085521"},     "shard1_0":{       "range":"80000000-bfffffff",       "state":"active",       "replicas":{"core_node66":{           "core":"hdfsbackuprestore_restored_shard1_0_replica_n65",           "base_url":"http://127.0.0.1:41321/solr",           "node_name":"127.0.0.1:41321_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265982814103467"}},   "router":{     "name":"compositeId",     "field":"shard_s"},   "maxShardsPerNode":"-1",   "autoAddReplicas":"true",   "nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:41321_solr has 3 replicas. Expected num replicas : 2. state: 
DocCollection(hdfsbackuprestore_restored//collections/hdfsbackuprestore_restored/state.json/11)={
  "pullReplicas":0,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node62":{
          "core":"hdfsbackuprestore_restored_shard2_replica_n61",
          "base_url":"http://127.0.0.1:41321/solr",
          "node_name":"127.0.0.1:41321_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265982814062346"},
    "shard1_1":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{"core_node64":{
          "core":"hdfsbackuprestore_restored_shard1_1_replica_n63",
          "base_url":"http://127.0.0.1:41321/solr",
          "node_name":"127.0.0.1:41321_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265982814085521"},
    "shard1_0":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{"core_node66":{
          "core":"hdfsbackuprestore_restored_shard1_0_replica_n65",
          "base_url":"http://127.0.0.1:41321/solr",
          "node_name":"127.0.0.1:41321_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265982814103467"}},
  "router":{
    "name":"compositeId",
    "field":"shard_s"},
  "maxShardsPerNode":"-1",
  "autoAddReplicas":"true",
  "nrtReplicas":1,
  "tlogReplicas":0}
	at __randomizedtesting.SeedInfo.seed([864380771B02AB2:803007DDDF4C474A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
	at java.base/java.util.HashMap.forEach(HashMap.java:1341)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
	at org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test(TestHdfsCloudBackupRestore.java:213)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test

Error Message:
Node 127.0.0.1:40289_solr has 3 replicas. Expected num replicas : 2. state:  DocCollection(hdfsbackuprestore_restored//collections/hdfsbackuprestore_restored/state.json/12)={   "pullReplicas":0,   "replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node62":{           "core":"hdfsbackuprestore_restored_shard2_replica_n61",           "base_url":"http://127.0.0.1:40289/solr",           "node_name":"127.0.0.1:40289_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265983046439427"},     "shard1_1":{       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{"core_node64":{           "core":"hdfsbackuprestore_restored_shard1_1_replica_n63",           "base_url":"http://127.0.0.1:40289/solr",           "node_name":"127.0.0.1:40289_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265983046481012"},     "shard1_0":{       "range":"80000000-bfffffff",       "state":"active",       "replicas":{"core_node66":{           "core":"hdfsbackuprestore_restored_shard1_0_replica_n65",           "base_url":"http://127.0.0.1:40289/solr",           "node_name":"127.0.0.1:40289_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265983046506317"}},   "router":{     "name":"compositeId",     "field":"shard_s"},   "maxShardsPerNode":"-1",   "autoAddReplicas":"true",   "nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:40289_solr has 3 replicas. Expected num replicas : 2. state: 
DocCollection(hdfsbackuprestore_restored//collections/hdfsbackuprestore_restored/state.json/12)={
  "pullReplicas":0,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node62":{
          "core":"hdfsbackuprestore_restored_shard2_replica_n61",
          "base_url":"http://127.0.0.1:40289/solr",
          "node_name":"127.0.0.1:40289_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265983046439427"},
    "shard1_1":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{"core_node64":{
          "core":"hdfsbackuprestore_restored_shard1_1_replica_n63",
          "base_url":"http://127.0.0.1:40289/solr",
          "node_name":"127.0.0.1:40289_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265983046481012"},
    "shard1_0":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{"core_node66":{
          "core":"hdfsbackuprestore_restored_shard1_0_replica_n65",
          "base_url":"http://127.0.0.1:40289/solr",
          "node_name":"127.0.0.1:40289_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265983046506317"}},
  "router":{
    "name":"compositeId",
    "field":"shard_s"},
  "maxShardsPerNode":"-1",
  "autoAddReplicas":"true",
  "nrtReplicas":1,
  "tlogReplicas":0}
	at __randomizedtesting.SeedInfo.seed([864380771B02AB2:803007DDDF4C474A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
	at java.base/java.util.HashMap.forEach(HashMap.java:1341)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
	at org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test(TestHdfsCloudBackupRestore.java:213)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test

Error Message:
Node 127.0.0.1:35683_solr has 3 replicas. Expected num replicas : 2. state:  DocCollection(hdfsbackuprestore_restored//collections/hdfsbackuprestore_restored/state.json/11)={   "pullReplicas":0,   "replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node62":{           "core":"hdfsbackuprestore_restored_shard2_replica_n61",           "base_url":"http://127.0.0.1:35683/solr",           "node_name":"127.0.0.1:35683_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542264615786561495"},     "shard1_1":{       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{"core_node64":{           "core":"hdfsbackuprestore_restored_shard1_1_replica_n63",           "base_url":"http://127.0.0.1:35683/solr",           "node_name":"127.0.0.1:35683_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542264615786580102"},     "shard1_0":{       "range":"80000000-bfffffff",       "state":"active",       "replicas":{"core_node66":{           "core":"hdfsbackuprestore_restored_shard1_0_replica_n65",           "base_url":"http://127.0.0.1:35683/solr",           "node_name":"127.0.0.1:35683_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542264615786596125"}},   "router":{     "name":"compositeId",     "field":"shard_s"},   "maxShardsPerNode":"-1",   "autoAddReplicas":"true",   "nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:35683_solr has 3 replicas. Expected num replicas : 2. state: 
DocCollection(hdfsbackuprestore_restored//collections/hdfsbackuprestore_restored/state.json/11)={
  "pullReplicas":0,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node62":{
          "core":"hdfsbackuprestore_restored_shard2_replica_n61",
          "base_url":"http://127.0.0.1:35683/solr",
          "node_name":"127.0.0.1:35683_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542264615786561495"},
    "shard1_1":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{"core_node64":{
          "core":"hdfsbackuprestore_restored_shard1_1_replica_n63",
          "base_url":"http://127.0.0.1:35683/solr",
          "node_name":"127.0.0.1:35683_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542264615786580102"},
    "shard1_0":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{"core_node66":{
          "core":"hdfsbackuprestore_restored_shard1_0_replica_n65",
          "base_url":"http://127.0.0.1:35683/solr",
          "node_name":"127.0.0.1:35683_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542264615786596125"}},
  "router":{
    "name":"compositeId",
    "field":"shard_s"},
  "maxShardsPerNode":"-1",
  "autoAddReplicas":"true",
  "nrtReplicas":1,
  "tlogReplicas":0}
	at __randomizedtesting.SeedInfo.seed([864380771B02AB2:803007DDDF4C474A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
	at java.base/java.util.HashMap.forEach(HashMap.java:1341)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
	at org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test(TestHdfsCloudBackupRestore.java:213)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test

Error Message:
Node 127.0.0.1:40471_solr has 3 replicas. Expected num replicas : 2. state:  DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/10)={   "pullReplicas":0,   "replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node62":{           "core":"backuprestore_restored_shard2_replica_n61",           "base_url":"https://127.0.0.1:40471/solr",           "node_name":"127.0.0.1:40471_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265967862948796"},     "shard1_1":{       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{"core_node64":{           "core":"backuprestore_restored_shard1_1_replica_n63",           "base_url":"https://127.0.0.1:40471/solr",           "node_name":"127.0.0.1:40471_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265967862987983"},     "shard1_0":{       "range":"80000000-bfffffff",       "state":"active",       "replicas":{"core_node66":{           "core":"backuprestore_restored_shard1_0_replica_n65",           "base_url":"https://127.0.0.1:40471/solr",           "node_name":"127.0.0.1:40471_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265967863018246"}},   "router":{     "name":"compositeId",     "field":"shard_s"},   "maxShardsPerNode":"-1",   "autoAddReplicas":"true",   "nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:40471_solr has 3 replicas. Expected num replicas : 2. state: 
DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/10)={
  "pullReplicas":0,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node62":{
          "core":"backuprestore_restored_shard2_replica_n61",
          "base_url":"https://127.0.0.1:40471/solr",
          "node_name":"127.0.0.1:40471_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265967862948796"},
    "shard1_1":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{"core_node64":{
          "core":"backuprestore_restored_shard1_1_replica_n63",
          "base_url":"https://127.0.0.1:40471/solr",
          "node_name":"127.0.0.1:40471_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265967862987983"},
    "shard1_0":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{"core_node66":{
          "core":"backuprestore_restored_shard1_0_replica_n65",
          "base_url":"https://127.0.0.1:40471/solr",
          "node_name":"127.0.0.1:40471_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265967863018246"}},
  "router":{
    "name":"compositeId",
    "field":"shard_s"},
  "maxShardsPerNode":"-1",
  "autoAddReplicas":"true",
  "nrtReplicas":1,
  "tlogReplicas":0}
	at __randomizedtesting.SeedInfo.seed([864380771B02AB2:803007DDDF4C474A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
	at java.base/java.util.HashMap.forEach(HashMap.java:1341)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
	at org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:64)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test

Error Message:
Node 127.0.0.1:38607_solr has 3 replicas. Expected num replicas : 2. state:  DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/10)={   "pullReplicas":0,   "replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node62":{           "core":"backuprestore_restored_shard2_replica_n61",           "base_url":"https://127.0.0.1:38607/solr",           "node_name":"127.0.0.1:38607_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265968599149488"},     "shard1_1":{       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{"core_node64":{           "core":"backuprestore_restored_shard1_1_replica_n63",           "base_url":"https://127.0.0.1:38607/solr",           "node_name":"127.0.0.1:38607_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265968599191569"},     "shard1_0":{       "range":"80000000-bfffffff",       "state":"active",       "replicas":{"core_node66":{           "core":"backuprestore_restored_shard1_0_replica_n65",           "base_url":"https://127.0.0.1:38607/solr",           "node_name":"127.0.0.1:38607_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265968599223132"}},   "router":{     "name":"compositeId",     "field":"shard_s"},   "maxShardsPerNode":"-1",   "autoAddReplicas":"true",   "nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:38607_solr has 3 replicas. Expected num replicas : 2. state: 
DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/10)={
  "pullReplicas":0,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node62":{
          "core":"backuprestore_restored_shard2_replica_n61",
          "base_url":"https://127.0.0.1:38607/solr",
          "node_name":"127.0.0.1:38607_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265968599149488"},
    "shard1_1":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{"core_node64":{
          "core":"backuprestore_restored_shard1_1_replica_n63",
          "base_url":"https://127.0.0.1:38607/solr",
          "node_name":"127.0.0.1:38607_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265968599191569"},
    "shard1_0":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{"core_node66":{
          "core":"backuprestore_restored_shard1_0_replica_n65",
          "base_url":"https://127.0.0.1:38607/solr",
          "node_name":"127.0.0.1:38607_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265968599223132"}},
  "router":{
    "name":"compositeId",
    "field":"shard_s"},
  "maxShardsPerNode":"-1",
  "autoAddReplicas":"true",
  "nrtReplicas":1,
  "tlogReplicas":0}
	at __randomizedtesting.SeedInfo.seed([864380771B02AB2:803007DDDF4C474A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
	at java.base/java.util.HashMap.forEach(HashMap.java:1341)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
	at org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:64)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test

Error Message:
Node 127.0.0.1:35503_solr has 3 replicas. Expected num replicas : 2. state:  DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/10)={   "pullReplicas":0,   "replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node62":{           "core":"backuprestore_restored_shard2_replica_n61",           "base_url":"https://127.0.0.1:35503/solr",           "node_name":"127.0.0.1:35503_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265982798866060"},     "shard1_1":{       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{"core_node64":{           "core":"backuprestore_restored_shard1_1_replica_n63",           "base_url":"https://127.0.0.1:35503/solr",           "node_name":"127.0.0.1:35503_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265982798902665"},     "shard1_0":{       "range":"80000000-bfffffff",       "state":"active",       "replicas":{"core_node66":{           "core":"backuprestore_restored_shard1_0_replica_n65",           "base_url":"https://127.0.0.1:35503/solr",           "node_name":"127.0.0.1:35503_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265982798925757"}},   "router":{     "name":"compositeId",     "field":"shard_s"},   "maxShardsPerNode":"-1",   "autoAddReplicas":"true",   "nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:35503_solr has 3 replicas. Expected num replicas : 2. state: 
DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/10)={
  "pullReplicas":0,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node62":{
          "core":"backuprestore_restored_shard2_replica_n61",
          "base_url":"https://127.0.0.1:35503/solr",
          "node_name":"127.0.0.1:35503_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265982798866060"},
    "shard1_1":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{"core_node64":{
          "core":"backuprestore_restored_shard1_1_replica_n63",
          "base_url":"https://127.0.0.1:35503/solr",
          "node_name":"127.0.0.1:35503_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265982798902665"},
    "shard1_0":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{"core_node66":{
          "core":"backuprestore_restored_shard1_0_replica_n65",
          "base_url":"https://127.0.0.1:35503/solr",
          "node_name":"127.0.0.1:35503_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265982798925757"}},
  "router":{
    "name":"compositeId",
    "field":"shard_s"},
  "maxShardsPerNode":"-1",
  "autoAddReplicas":"true",
  "nrtReplicas":1,
  "tlogReplicas":0}
	at __randomizedtesting.SeedInfo.seed([864380771B02AB2:803007DDDF4C474A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
	at java.base/java.util.HashMap.forEach(HashMap.java:1341)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
	at org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:64)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test

Error Message:
Node 127.0.0.1:42365_solr has 3 replicas. Expected num replicas : 2. state:  DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/10)={   "pullReplicas":0,   "replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node62":{           "core":"backuprestore_restored_shard2_replica_n61",           "base_url":"https://127.0.0.1:42365/solr",           "node_name":"127.0.0.1:42365_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265996233552252"},     "shard1_1":{       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{"core_node64":{           "core":"backuprestore_restored_shard1_1_replica_n63",           "base_url":"https://127.0.0.1:42365/solr",           "node_name":"127.0.0.1:42365_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265996233583183"},     "shard1_0":{       "range":"80000000-bfffffff",       "state":"active",       "replicas":{"core_node66":{           "core":"backuprestore_restored_shard1_0_replica_n65",           "base_url":"https://127.0.0.1:42365/solr",           "node_name":"127.0.0.1:42365_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265996233612092"}},   "router":{     "name":"compositeId",     "field":"shard_s"},   "maxShardsPerNode":"-1",   "autoAddReplicas":"true",   "nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:42365_solr has 3 replicas. Expected num replicas : 2. state: 
DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/10)={
  "pullReplicas":0,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node62":{
          "core":"backuprestore_restored_shard2_replica_n61",
          "base_url":"https://127.0.0.1:42365/solr",
          "node_name":"127.0.0.1:42365_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265996233552252"},
    "shard1_1":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{"core_node64":{
          "core":"backuprestore_restored_shard1_1_replica_n63",
          "base_url":"https://127.0.0.1:42365/solr",
          "node_name":"127.0.0.1:42365_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265996233583183"},
    "shard1_0":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{"core_node66":{
          "core":"backuprestore_restored_shard1_0_replica_n65",
          "base_url":"https://127.0.0.1:42365/solr",
          "node_name":"127.0.0.1:42365_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265996233612092"}},
  "router":{
    "name":"compositeId",
    "field":"shard_s"},
  "maxShardsPerNode":"-1",
  "autoAddReplicas":"true",
  "nrtReplicas":1,
  "tlogReplicas":0}
	at __randomizedtesting.SeedInfo.seed([864380771B02AB2:803007DDDF4C474A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
	at java.base/java.util.HashMap.forEach(HashMap.java:1341)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
	at org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:64)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test

Error Message:
Node 127.0.0.1:35363_solr has 3 replicas. Expected num replicas : 2. state:  DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/11)={   "pullReplicas":0,   "replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node62":{           "core":"backuprestore_restored_shard2_replica_n61",           "base_url":"https://127.0.0.1:35363/solr",           "node_name":"127.0.0.1:35363_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265937934565781"},     "shard1_1":{       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{"core_node64":{           "core":"backuprestore_restored_shard1_1_replica_n63",           "base_url":"https://127.0.0.1:35363/solr",           "node_name":"127.0.0.1:35363_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265937934619654"},     "shard1_0":{       "range":"80000000-bfffffff",       "state":"active",       "replicas":{"core_node66":{           "core":"backuprestore_restored_shard1_0_replica_n65",           "base_url":"https://127.0.0.1:35363/solr",           "node_name":"127.0.0.1:35363_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542265937934660216"}},   "router":{     "name":"compositeId",     "field":"shard_s"},   "maxShardsPerNode":"-1",   "autoAddReplicas":"true",   "nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:35363_solr has 3 replicas. Expected num replicas : 2. state: 
DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/11)={
  "pullReplicas":0,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node62":{
          "core":"backuprestore_restored_shard2_replica_n61",
          "base_url":"https://127.0.0.1:35363/solr",
          "node_name":"127.0.0.1:35363_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265937934565781"},
    "shard1_1":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{"core_node64":{
          "core":"backuprestore_restored_shard1_1_replica_n63",
          "base_url":"https://127.0.0.1:35363/solr",
          "node_name":"127.0.0.1:35363_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265937934619654"},
    "shard1_0":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{"core_node66":{
          "core":"backuprestore_restored_shard1_0_replica_n65",
          "base_url":"https://127.0.0.1:35363/solr",
          "node_name":"127.0.0.1:35363_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542265937934660216"}},
  "router":{
    "name":"compositeId",
    "field":"shard_s"},
  "maxShardsPerNode":"-1",
  "autoAddReplicas":"true",
  "nrtReplicas":1,
  "tlogReplicas":0}
	at __randomizedtesting.SeedInfo.seed([864380771B02AB2:803007DDDF4C474A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
	at java.base/java.util.HashMap.forEach(HashMap.java:1341)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
	at org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:64)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test

Error Message:
Node 127.0.0.1:35465_solr has 3 replicas. Expected num replicas : 2. state:  DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/10)={   "pullReplicas":0,   "replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node62":{           "core":"backuprestore_restored_shard2_replica_n61",           "base_url":"https://127.0.0.1:35465/solr",           "node_name":"127.0.0.1:35465_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542264620086216682"},     "shard1_1":{       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{"core_node64":{           "core":"backuprestore_restored_shard1_1_replica_n63",           "base_url":"https://127.0.0.1:35465/solr",           "node_name":"127.0.0.1:35465_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542264620086268170"},     "shard1_0":{       "range":"80000000-bfffffff",       "state":"active",       "replicas":{"core_node66":{           "core":"backuprestore_restored_shard1_0_replica_n65",           "base_url":"https://127.0.0.1:35465/solr",           "node_name":"127.0.0.1:35465_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1542264620086290083"}},   "router":{     "name":"compositeId",     "field":"shard_s"},   "maxShardsPerNode":"-1",   "autoAddReplicas":"true",   "nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:35465_solr has 3 replicas. Expected num replicas : 2. state: 
DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/10)={
  "pullReplicas":0,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node62":{
          "core":"backuprestore_restored_shard2_replica_n61",
          "base_url":"https://127.0.0.1:35465/solr",
          "node_name":"127.0.0.1:35465_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542264620086216682"},
    "shard1_1":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{"core_node64":{
          "core":"backuprestore_restored_shard1_1_replica_n63",
          "base_url":"https://127.0.0.1:35465/solr",
          "node_name":"127.0.0.1:35465_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542264620086268170"},
    "shard1_0":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{"core_node66":{
          "core":"backuprestore_restored_shard1_0_replica_n65",
          "base_url":"https://127.0.0.1:35465/solr",
          "node_name":"127.0.0.1:35465_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1542264620086290083"}},
  "router":{
    "name":"compositeId",
    "field":"shard_s"},
  "maxShardsPerNode":"-1",
  "autoAddReplicas":"true",
  "nrtReplicas":1,
  "tlogReplicas":0}
	at __randomizedtesting.SeedInfo.seed([864380771B02AB2:803007DDDF4C474A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
	at java.base/java.util.HashMap.forEach(HashMap.java:1341)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
	at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
	at org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:64)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 14505 lines...]
   [junit4] Suite: org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore
   [junit4]   2> 1834081 INFO  (SUITE-TestLocalFSCloudBackupRestore-seed#[864380771B02AB2]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.api.collections.TestLocalFSCloudBackupRestore_864380771B02AB2-001/init-core-data-001
   [junit4]   2> 1834081 WARN  (SUITE-TestLocalFSCloudBackupRestore-seed#[864380771B02AB2]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=16 numCloses=16
   [junit4]   2> 1834082 INFO  (SUITE-TestLocalFSCloudBackupRestore-seed#[864380771B02AB2]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1834082 INFO  (SUITE-TestLocalFSCloudBackupRestore-seed#[864380771B02AB2]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
   [junit4]   2> 1834082 INFO  (SUITE-TestLocalFSCloudBackupRestore-seed#[864380771B02AB2]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cl

[...truncated too long message...]

ca_n61] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.backuprestore_restored.shard2.leader, tag=39077fc2
   [junit4]   2> 78836 INFO  (coreCloseExecutor-241-thread-3) [n:127.0.0.1:42365_solr c:backuprestore_restored s:shard1_1 r:core_node64 x:backuprestore_restored_shard1_1_replica_n63] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.backuprestore_restored.shard1_1.replica_n63, tag=5eb71aa4
   [junit4]   2> 78836 INFO  (coreCloseExecutor-241-thread-3) [n:127.0.0.1:42365_solr c:backuprestore_restored s:shard1_1 r:core_node64 x:backuprestore_restored_shard1_1_replica_n63] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@485b3b8a: rootName = solr_42365, domain = solr.core.backuprestore_restored.shard1_1.replica_n63, service url = null, agent id = null] for registry solr.core.backuprestore_restored.shard1_1.replica_n63 / com.codahale.metrics.MetricRegistry@fd9a771
   [junit4]   2> 78842 INFO  (coreCloseExecutor-242-thread-3) [n:127.0.0.1:37787_solr c:backuprestore s:shard1_1 r:core_node8 x:backuprestore_shard1_1_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.backuprestore.shard1_1.replica_n6, tag=7db53f3b
   [junit4]   2> 78842 INFO  (coreCloseExecutor-242-thread-3) [n:127.0.0.1:37787_solr c:backuprestore s:shard1_1 r:core_node8 x:backuprestore_shard1_1_replica_n6] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6e6354a9: rootName = solr_37787, domain = solr.core.backuprestore.shard1_1.replica_n6, service url = null, agent id = null] for registry solr.core.backuprestore.shard1_1.replica_n6 / com.codahale.metrics.MetricRegistry@47f38a6d
   [junit4]   2> 78844 INFO  (coreCloseExecutor-241-thread-4) [n:127.0.0.1:42365_solr c:backuprestore_restored s:shard1_0 r:core_node66 x:backuprestore_restored_shard1_0_replica_n65] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.backuprestore_restored.shard1_0.replica_n65, tag=15308c27
   [junit4]   2> 78844 INFO  (coreCloseExecutor-241-thread-4) [n:127.0.0.1:42365_solr c:backuprestore_restored s:shard1_0 r:core_node66 x:backuprestore_restored_shard1_0_replica_n65] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@74d35b9d: rootName = solr_42365, domain = solr.core.backuprestore_restored.shard1_0.replica_n65, service url = null, agent id = null] for registry solr.core.backuprestore_restored.shard1_0.replica_n65 / com.codahale.metrics.MetricRegistry@2c533f5d
   [junit4]   2> 78849 INFO  (coreCloseExecutor-241-thread-3) [n:127.0.0.1:42365_solr c:backuprestore_restored s:shard1_1 r:core_node64 x:backuprestore_restored_shard1_1_replica_n63] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.backuprestore_restored.shard1_1.leader, tag=5eb71aa4
   [junit4]   2> 78849 INFO  (coreCloseExecutor-241-thread-4) [n:127.0.0.1:42365_solr c:backuprestore_restored s:shard1_0 r:core_node66 x:backuprestore_restored_shard1_0_replica_n65] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.backuprestore_restored.shard1_0.leader, tag=15308c27
   [junit4]   2> 78851 INFO  (coreCloseExecutor-242-thread-1) [n:127.0.0.1:37787_solr c:backuprestore s:shard1 r:core_node2 x:backuprestore_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.backuprestore.shard1.leader, tag=5257d458
   [junit4]   2> 78851 INFO  (coreCloseExecutor-242-thread-2) [n:127.0.0.1:37787_solr c:backuprestore s:shard1_0 r:core_node7 x:backuprestore_shard1_0_replica_n5] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.backuprestore.shard1_0.leader, tag=2c76869
   [junit4]   2> 78851 INFO  (coreCloseExecutor-242-thread-3) [n:127.0.0.1:37787_solr c:backuprestore s:shard1_1 r:core_node8 x:backuprestore_shard1_1_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.backuprestore.shard1_1.leader, tag=7db53f3b
   [junit4]   2> 78854 INFO  (jetty-closer-131-thread-1) [    ] o.a.s.c.Overseer Overseer (id=72165088549994500-127.0.0.1:42365_solr-n_0000000000) closing
   [junit4]   2> 78854 INFO  (OverseerStateUpdate-72165088549994500-127.0.0.1:42365_solr-n_0000000000) [n:127.0.0.1:42365_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:42365_solr
   [junit4]   2> 78854 WARN  (OverseerAutoScalingTriggerThread-72165088549994500-127.0.0.1:42365_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 78856 INFO  (jetty-closer-131-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2ac5383c{/solr,null,UNAVAILABLE}
   [junit4]   2> 78856 INFO  (jetty-closer-131-thread-2) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 78857 INFO  (jetty-closer-131-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5f07aba6{/solr,null,UNAVAILABLE}
   [junit4]   2> 78857 INFO  (jetty-closer-131-thread-1) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 78858 ERROR (SUITE-TestLocalFSCloudBackupRestore-seed#[864380771B02AB2]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 78858 INFO  (SUITE-TestLocalFSCloudBackupRestore-seed#[864380771B02AB2]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46837 46837
   [junit4]   2> 78979 INFO  (Thread-321) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46837 46837
   [junit4]   2> 78980 WARN  (Thread-321) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/aliases.json
   [junit4]   2> 	3	/solr/collections/backuprestore/terms/shard1_0
   [junit4]   2> 	3	/solr/collections/backuprestore_restored/terms/shard1_0
   [junit4]   2> 	3	/solr/collections/backuprestore_restored/terms/shard1_1
   [junit4]   2> 	3	/solr/collections/backuprestore/terms/shard1_1
   [junit4]   2> 	3	/solr/collections/backuprestore/terms/shard2
   [junit4]   2> 	3	/solr/collections/backuprestore_restored/terms/shard2
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 	2	/solr/configs/conf1
   [junit4]   2> 	2	/solr/collections/backuprestore/terms/shard1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/clusterstate.json
   [junit4]   2> 	5	/solr/clusterprops.json
   [junit4]   2> 	2	/solr/collections/backuprestore_restored/state.json
   [junit4]   2> 	2	/solr/collections/backuprestore/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/live_nodes
   [junit4]   2> 	5	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.api.collections.TestLocalFSCloudBackupRestore_864380771B02AB2-003
   [junit4]   2> Nov 15, 2018 7:13:16 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {}, locale=fur, timezone=Asia/Bahrain
   [junit4]   2> NOTE: Linux 4.15.0-38-generic amd64/Oracle Corporation 10.0.1 (64-bit)/cpus=8,threads=1,free=277653072,total=457179136
   [junit4]   2> NOTE: All tests run in this JVM: [TestHdfsCloudBackupRestore, TestLocalFSCloudBackupRestore, TestLocalFSCloudBackupRestore, TestLocalFSCloudBackupRestore]
   [junit4] Completed [10/10 (10!)] on J1 in 13.10s, 1 test, 1 failure <<< FAILURES!

[...truncated 20 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/common-build.xml:1567: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/common-build.xml:1092: There were test failures: 10 suites, 10 tests, 10 failures [seed: 864380771B02AB2]

Total time: 1 minute 21 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   5/5 failed: org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore
[repro]   5/5 failed: org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=c4961d48dfb66223977ce6678f6de94f93f841c9, workspace=/var/lib/jenkins/workspace/Lucene-Solr-7.x-Linux
[WARNINGS] Computing warning deltas based on reference build #3094
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-9.0.4) - Build # 3094 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3094/
Java: 64bit/jdk-9.0.4 -XX:-UseCompressedOops -XX:+UseG1GC

4 tests failed.
FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:40903/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:40903/solr
	at __randomizedtesting.SeedInfo.seed([EF1D694F57FF56D4:2EED10E37AAF9C73]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
	at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:39887/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:39887/solr
	at __randomizedtesting.SeedInfo.seed([EF1D694F57FF56D4:2EED10E37AAF9C73]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
	at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:40353/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:40353/solr
	at __randomizedtesting.SeedInfo.seed([EF1D694F57FF56D4:2EED10E37AAF9C73]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
	at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:37263/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:37263/solr
	at __randomizedtesting.SeedInfo.seed([EF1D694F57FF56D4:2EED10E37AAF9C73]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
	at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 14358 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestTlogReplica
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestTlogReplica_EF1D694F57FF56D4-001/init-core-data-001
   [junit4]   2> 1782522 INFO  (SUITE-TestTlogReplica-seed#[EF1D694F57FF56D4]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestTlogReplica_EF1D694F57FF56D4-001/tempDir-001
   [junit4]   2> 1782522 INFO  (SUITE-TestTlogReplica-seed#[EF1D694F57FF56D4]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1782530 INFO  (Thread-5405) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1782530 INFO  (Thread-5405) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1782549 ERROR (Thread-5405) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1782630 INFO  (SUITE-TestTlogReplica-seed#[EF1D694F57FF56D4]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:35163
   [junit4]   2> 1782631 INFO  (zkConnectionManagerCallback-6231-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1782632 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10060e3e96f0000, likely client has closed socket
   [junit4]   2> 1782633 INFO  (jetty-launcher-6228-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 9.0.4+11
   [junit4]   2> 1782634 INFO  (jetty-launcher-6228-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 9.0.4+11
   [junit4]   2> 1782661 INFO  (jetty-launcher-6228-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1782661 INFO  (jetty-launcher-6228-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1782661 INFO  (jetty-launcher-6228-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1782661 INFO  (jetty-launcher-6228-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1782661 INFO  (jetty-launcher-6228-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1782661 INFO  (jetty-launcher-6228-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1782662 INFO  (jetty-launcher-6228-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2aea7ff8{/solr,null,AVAILABLE}
   [junit4]   2> 1782662 INFO  (jetty-launcher-6228-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@69b920a7{/solr,null,AVAILABLE}
   [junit4]   2> 1782662 INFO  (jetty-launcher-6228-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@55f2a932{SSL,[ssl, http/1.1]}{127.0.0.1:34977}
   [junit4]   2> 1782662 INFO  (jetty-launcher-6228-thread-1) [    ] o.e.j.s.Server Started @1782753ms
   [junit4]   2> 1782662 INFO  (jetty-launcher-6228-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@60ff6932{SSL,[ssl, http/1.1]}{127.0.0.1:37263}
   [junit4]   2> 1782662 INFO  (jetty-launcher-6228-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34977}
   [junit4]   2> 1782662 INFO  (jetty-launcher-6228-thread-2) [    ] o.e.j.s.Server Started @1782753ms
   [junit4]   2> 1782662 INFO  (jetty-launcher-6228-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37263}
   [junit4]   2> 1782663 ERROR (jetty-launcher-6228-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1782663 INFO  (jetty-launcher-6228-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1782663 ERROR (jetty-launcher-6228-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1782663 INFO  (jetty-launcher-6228-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1782663 INFO  (jetty-launcher-6228-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 1782663 INFO  (jetty-launcher-6228-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 1782663 INFO  (jetty-launcher-6228-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1782663 INFO  (jetty-launcher-6228-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1782663 INFO  (jetty-launcher-6228-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1782663 INFO  (jetty-launcher-6228-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1782663 INFO  (jetty-launcher-6228-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-15T03:08:16.946202Z
   [junit4]   2> 1782663 INFO  (jetty-launcher-6228-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-15T03:08:16.946212Z
   [junit4]   2> 1782673 INFO  (zkConnectionManagerCallback-6233-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1782673 INFO  (zkConnectionManagerCallback-6235-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1782674 INFO  (jetty-launcher-6228-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1782674 INFO  (jetty-launcher-6228-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1782764 INFO  (jetty-launcher-6228-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35163/solr
   [junit4]   2> 1782765 INFO  (zkConnectionManagerCallback-6239-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1782766 INFO  (zkConnectionManagerCallback-6241-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1782808 INFO  (jetty-launcher-6228-thread-2) [n:127.0.0.1:37263_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37263_solr
   [junit4]   2> 1782809 INFO  (jetty-launcher-6228-thread-2) [n:127.0.0.1:37263_solr    ] o.a.s.c.Overseer Overseer (id=72164126028136452-127.0.0.1:37263_solr-n_0000000000) starting
   [junit4]   2> 1782811 INFO  (zkConnectionManagerCallback-6248-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1782812 INFO  (jetty-launcher-6228-thread-2) [n:127.0.0.1:37263_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35163/solr ready
   [junit4]   2> 1782812 INFO  (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37263_solr
   [junit4]   2> 1782821 INFO  (jetty-launcher-6228-thread-2) [n:127.0.0.1:37263_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37263_solr
   [junit4]   2> 1782822 INFO  (zkCallback-6247-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1782838 INFO  (zkCallback-6240-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1782847 DEBUG (OverseerAutoScalingTriggerThread-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:37263_solr]
   [junit4]   2> 1782853 DEBUG (ScheduledTrigger-9490-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 1782857 INFO  (jetty-launcher-6228-thread-2) [n:127.0.0.1:37263_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1782871 INFO  (jetty-launcher-6228-thread-2) [n:127.0.0.1:37263_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37263.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@613c67f3
   [junit4]   2> 1782880 INFO  (jetty-launcher-6228-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35163/solr
   [junit4]   2> 1782881 INFO  (zkConnectionManagerCallback-6253-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1782881 INFO  (jetty-launcher-6228-thread-2) [n:127.0.0.1:37263_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37263.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@613c67f3
   [junit4]   2> 1782881 INFO  (jetty-launcher-6228-thread-2) [n:127.0.0.1:37263_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37263.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@613c67f3
   [junit4]   2> 1782882 INFO  (jetty-launcher-6228-thread-2) [n:127.0.0.1:37263_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestTlogReplica_EF1D694F57FF56D4-001/tempDir-001/node2/.
   [junit4]   2> 1782885 INFO  (zkConnectionManagerCallback-6255-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1782890 INFO  (jetty-launcher-6228-thread-1) [n:127.0.0.1:34977_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1782892 INFO  (jetty-launcher-6228-thread-1) [n:127.0.0.1:34977_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1782892 INFO  (jetty-launcher-6228-thread-1) [n:127.0.0.1:34977_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34977_solr
   [junit4]   2> 1782892 DEBUG (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:34977_solr"} current state version: 0
   [junit4]   2> 1782892 INFO  (zkCallback-6247-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1782892 INFO  (zkCallback-6240-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1782897 INFO  (zkCallback-6254-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1782904 INFO  (zkConnectionManagerCallback-6262-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1782905 INFO  (jetty-launcher-6228-thread-1) [n:127.0.0.1:34977_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1782906 INFO  (jetty-launcher-6228-thread-1) [n:127.0.0.1:34977_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35163/solr ready
   [junit4]   2> 1782907 INFO  (jetty-launcher-6228-thread-1) [n:127.0.0.1:34977_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1782919 INFO  (jetty-launcher-6228-thread-1) [n:127.0.0.1:34977_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34977.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@613c67f3
   [junit4]   2> 1782926 INFO  (jetty-launcher-6228-thread-1) [n:127.0.0.1:34977_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34977.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@613c67f3
   [junit4]   2> 1782926 INFO  (jetty-launcher-6228-thread-1) [n:127.0.0.1:34977_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34977.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@613c67f3
   [junit4]   2> 1782926 INFO  (jetty-launcher-6228-thread-1) [n:127.0.0.1:34977_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestTlogReplica_EF1D694F57FF56D4-001/tempDir-001/node1/.
   [junit4]   2> 1782945 INFO  (zkConnectionManagerCallback-6265-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1782947 INFO  (zkConnectionManagerCallback-6270-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1782947 INFO  (SUITE-TestTlogReplica-seed#[EF1D694F57FF56D4]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1782948 INFO  (SUITE-TestTlogReplica-seed#[EF1D694F57FF56D4]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35163/solr ready
   [junit4]   2> 1782949 INFO  (SUITE-TestTlogReplica-seed#[EF1D694F57FF56D4]-worker) [    ] o.a.s.c.TestTlogReplica Using legacyCloud?: false
   [junit4]   2> 1782959 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with params val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1782960 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1782977 INFO  (qtp1168330160-24477) [n:127.0.0.1:37263_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_recovery&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1782979 INFO  (OverseerThreadFactory-9492-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection tlog_replica_test_recovery
   [junit4]   2> 1782982 DEBUG (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "name":"tlog_replica_test_recovery",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"conf",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "maxShardsPerNode":"100",
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "tlogReplicas":"2",
   [junit4]   2>   "nrtReplicas":"0",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "replicationFactor":"0",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 1783086 INFO  (qtp1168330160-24467) [n:127.0.0.1:37263_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1783094 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1783096 DEBUG (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_recovery",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_recovery_shard1_replica_t1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:34977/solr",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 1783096 INFO  (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_recovery",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_recovery_shard1_replica_t1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:34977/solr",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 1783097 DEBUG (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_recovery",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_recovery_shard1_replica_t2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:37263/solr",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 1783097 INFO  (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_recovery",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_recovery_shard1_replica_t2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:37263/solr",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 1783300 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr    x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_recovery_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_recovery&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 1783300 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr    x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_recovery_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_recovery&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 1783300 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr    x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1783302 DEBUG (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "core":"tlog_replica_test_recovery_shard1_replica_t1",
   [junit4]   2>   "core_node_name":"core_node3",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:34977/solr",
   [junit4]   2>   "node_name":"127.0.0.1:34977_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"tlog_replica_test_recovery",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "force_set_state":"false",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1783302 DEBUG (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "core":"tlog_replica_test_recovery_shard1_replica_t2",
   [junit4]   2>   "core_node_name":"core_node4",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:37263/solr",
   [junit4]   2>   "node_name":"127.0.0.1:37263_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"tlog_replica_test_recovery",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "force_set_state":"false",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1783854 DEBUG (ScheduledTrigger-9490-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1784306 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 1784306 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 1784311 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.s.IndexSchema [tlog_replica_test_recovery_shard1_replica_t1] Schema name=minimal
   [junit4]   2> 1784311 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.s.IndexSchema [tlog_replica_test_recovery_shard1_replica_t2] Schema name=minimal
   [junit4]   2> 1784313 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1784313 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1784313 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_recovery_shard1_replica_t1' using configuration from collection tlog_replica_test_recovery, trusted=true
   [junit4]   2> 1784313 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_recovery_shard1_replica_t2' using configuration from collection tlog_replica_test_recovery, trusted=true
   [junit4]   2> 1784313 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37263.solr.core.tlog_replica_test_recovery.shard1.replica_t2' (registry 'solr.core.tlog_replica_test_recovery.shard1.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@613c67f3
   [junit4]   2> 1784313 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34977.solr.core.tlog_replica_test_recovery.shard1.replica_t1' (registry 'solr.core.tlog_replica_test_recovery.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@613c67f3
   [junit4]   2> 1784313 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.SolrCore [[tlog_replica_test_recovery_shard1_replica_t2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestTlogReplica_EF1D694F57FF56D4-001/tempDir-001/node2/tlog_replica_test_recovery_shard1_replica_t2], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestTlogReplica_EF1D694F57FF56D4-001/tempDir-001/node2/./tlog_replica_test_recovery_shard1_replica_t2/data/]
   [junit4]   2> 1784313 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.SolrCore [[tlog_replica_test_recovery_shard1_replica_t1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestTlogReplica_EF1D694F57FF56D4-001/tempDir-001/node1/tlog_replica_test_recovery_shard1_replica_t1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestTlogReplica_EF1D694F57FF56D4-001/tempDir-001/node1/./tlog_replica_test_recovery_shard1_replica_t1/data/]
   [junit4]   2> 1784365 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1784365 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1784366 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1784366 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1784368 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3ec422b4[tlog_replica_test_recovery_shard1_replica_t1] main]
   [junit4]   2> 1784368 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1784368 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1784369 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1784369 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1784369 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1784370 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1617167697736368128
   [junit4]   2> 1784370 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1784370 INFO  (searcherExecutor-9504-thread-1-processing-n:127.0.0.1:34977_solr x:tlog_replica_test_recovery_shard1_replica_t1 c:tlog_replica_test_recovery s:shard1 r:core_node3) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.SolrCore [tlog_replica_test_recovery_shard1_replica_t1] Registered new searcher Searcher@3ec422b4[tlog_replica_test_recovery_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1784370 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1784372 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@1d39a63[tlog_replica_test_recovery_shard1_replica_t2] main]
   [junit4]   2> 1784373 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tlog_replica_test_recovery/terms/shard1 to Terms{values={core_node3=0}, version=0}
   [junit4]   2> 1784376 DEBUG (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"tlog_replica_test_recovery"} current state version: 0
   [junit4]   2> 1784376 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 1784375 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1784376 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1784376 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1784377 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1617167697743708160
   [junit4]   2> 1784377 INFO  (searcherExecutor-9505-thread-1-processing-n:127.0.0.1:37263_solr x:tlog_replica_test_recovery_shard1_replica_t2 c:tlog_replica_test_recovery s:shard1 r:core_node4) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.SolrCore [tlog_replica_test_recovery_shard1_replica_t2] Registered new searcher Searcher@1d39a63[tlog_replica_test_recovery_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1784380 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tlog_replica_test_recovery/terms/shard1 to Terms{values={core_node3=0, core_node4=0}, version=1}
   [junit4]   2> 1784855 DEBUG (ScheduledTrigger-9490-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1784876 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1784876 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1784876 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:34977/solr/tlog_replica_test_recovery_shard1_replica_t1/
   [junit4]   2> 1784877 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_recovery_shard1_replica_t1 url=https://127.0.0.1:34977/solr START replicas=[https://127.0.0.1:37263/solr/tlog_replica_test_recovery_shard1_replica_t2/] nUpdates=100
   [junit4]   2> 1784877 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_recovery_shard1_replica_t1 url=https://127.0.0.1:34977/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 1784879 INFO  (qtp1168330160-24540) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_recovery_shard1_replica_t2]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1784879 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 1784879 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1784879 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1784880 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.ZkController tlog_replica_test_recovery_shard1_replica_t1 stopping background replication from leader
   [junit4]   2> 1784881 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:34977/solr/tlog_replica_test_recovery_shard1_replica_t1/ shard1
   [junit4]   2> 1784882 DEBUG (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"tlog_replica_test_recovery",
   [junit4]   2>   "base_url":"https://127.0.0.1:34977/solr",
   [junit4]   2>   "core":"tlog_replica_test_recovery_shard1_replica_t1",
   [junit4]   2>   "state":"active"} current state version: 0
   [junit4]   2> 1784983 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1784983 DEBUG (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"tlog_replica_test_recovery_shard1_replica_t1",
   [junit4]   2>   "core_node_name":"core_node3",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:34977/solr",
   [junit4]   2>   "node_name":"127.0.0.1:34977_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"tlog_replica_test_recovery",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "force_set_state":"false",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1784984 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_recovery_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_recovery&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1683
   [junit4]   2> 1785084 INFO  (zkCallback-6254-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1785382 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.ZkController tlog_replica_test_recovery_shard1_replica_t2 starting background replication from leader
   [junit4]   2> 1785382 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:01
   [junit4]   2> 1785383 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 1785383 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1785383 INFO  (indexFetcher-9515-thread-1) [    ] o.a.s.h.IndexFetcher Updated masterUrl to https://127.0.0.1:34977/solr/tlog_replica_test_recovery_shard1_replica_t1/
   [junit4]   2> 1785384 INFO  (qtp254001560-24472) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_recovery_shard1_replica_t1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1785384 DEBUG (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"tlog_replica_test_recovery_shard1_replica_t2",
   [junit4]   2>   "core_node_name":"core_node4",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:37263/solr",
   [junit4]   2>   "node_name":"127.0.0.1:37263_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"tlog_replica_test_recovery",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "force_set_state":"false",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1785384 INFO  (indexFetcher-9515-thread-1) [    ] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 1785384 INFO  (indexFetcher-9515-thread-1) [    ] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 1785384 INFO  (indexFetcher-9515-thread-1) [    ] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1785385 INFO  (indexFetcher-9515-thread-1) [    ] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 1785385 INFO  (indexFetcher-9515-thread-1) [    ] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
   [junit4]   2> 1785385 INFO  (indexFetcher-9515-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@35b8786f[tlog_replica_test_recovery_shard1_replica_t2] main]
   [junit4]   2> 1785386 INFO  (searcherExecutor-9505-thread-1) [    ] o.a.s.c.SolrCore [tlog_replica_test_recovery_shard1_replica_t2] Registered new searcher Searcher@35b8786f[tlog_replica_test_recovery_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1785412 INFO  (qtp1168330160-24475) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_recovery_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_recovery&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2112
   [junit4]   2> 1785415 INFO  (qtp1168330160-24477) [n:127.0.0.1:37263_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1785485 INFO  (zkCallback-6254-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1785485 INFO  (zkCallback-6240-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1785855 DEBUG (ScheduledTrigger-9490-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1786383 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_recovery_shard1_replica_t1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1786384 INFO  (indexFetcher-9515-thread-1) [    ] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 1786384 INFO  (indexFetcher-9515-thread-1) [    ] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 1786384 INFO  (indexFetcher-9515-thread-1) [    ] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1786384 INFO  (indexFetcher-9515-thread-1) [    ] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 1786384 INFO  (indexFetcher-9515-thread-1) [    ] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
   [junit4]   2> 1786384 INFO  (indexFetcher-9515-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@333699a5[tlog_replica_test_recovery_shard1_replica_t2] main]
   [junit4]   2> 1786385 INFO  (searcherExecutor-9505-thread-1) [    ] o.a.s.c.SolrCore [tlog_replica_test_recovery_shard1_replica_t2] Registered new searcher Searcher@333699a5[tlog_replica_test_recovery_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1786415 INFO  (qtp1168330160-24477) [n:127.0.0.1:37263_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_recovery&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2} status=0 QTime=3437
   [junit4]   2> 1786420 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tlog_replica_test_recovery/terms/shard1 to Terms{values={core_node3=1, core_node4=1}, version=2}
   [junit4]   2> 1786422 INFO  (qtp1168330160-24477) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:34977/solr/tlog_replica_test_recovery_shard1_replica_t1/&wt=javabin&version=2}{add=[3 (1617167699884900352), 4 (1617167699884900353)]} 0 0
   [junit4]   2> 1786422 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t1]  webapp=/solr path=/update params={_stateVer_=tlog_replica_test_recovery:6&wt=javabin&version=2}{add=[3 (1617167699884900352), 4 (1617167699884900353)]} 0 3
   [junit4]   2> 1786424 INFO  (qtp254001560-24466) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1617167699890143232,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1786425 INFO  (qtp254001560-24466) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6c24d8a3 commitCommandVersion:1617167699890143232
   [junit4]   2> 1786427 INFO  (qtp1168330160-24538) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:34977/solr/tlog_replica_test_recovery_shard1_replica_t1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
   [junit4]   2> 1786427 INFO  (qtp254001560-24466) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3e10f563[tlog_replica_test_recovery_shard1_replica_t1] main]
   [junit4]   2> 1786427 INFO  (qtp254001560-24466) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1786429 INFO  (searcherExecutor-9504-thread-1-processing-n:127.0.0.1:34977_solr x:tlog_replica_test_recovery_shard1_replica_t1 c:tlog_replica_test_recovery s:shard1 r:core_node3) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.SolrCore [tlog_replica_test_recovery_shard1_replica_t1] Registered new searcher Searcher@3e10f563[tlog_replica_test_recovery_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader(_0(7.7.0):C2))}
   [junit4]   2> 1786429 INFO  (qtp254001560-24466) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:34977/solr/tlog_replica_test_recovery_shard1_replica_t1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 4
   [junit4]   2> 1786429 INFO  (qtp254001560-24546) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t1]  webapp=/solr path=/update params={_stateVer_=tlog_replica_test_recovery:6&commit=true&wt=javabin&version=2}{commit=} 0 6
   [junit4]   2> 1786431 INFO  (qtp1168330160-24477) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:34977/solr/tlog_replica_test_recovery_shard1_replica_t1/&wt=javabin&version=2}{add=[5 (1617167699896434688)]} 0 0
   [junit4]   2> 1786431 INFO  (qtp254001560-24474) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[5 (1617167699896434688)]} 0 1
   [junit4]   2> 1786432 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.c.ChaosMonkey monkey: kill jetty! 37263
   [junit4]   2> 1786437 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@60ff6932{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1786438 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=585086106
   [junit4]   2> 1786438 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1786438 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@71881e7c: rootName = solr_37263, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@4b38df9b
   [junit4]   2> 1786444 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1786444 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@736131c8: rootName = solr_37263, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@4a275658
   [junit4]   2> 1786447 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1786447 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3f114fd8: rootName = solr_37263, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@2298af5d
   [junit4]   2> 1786448 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:37263_solr
   [junit4]   2> 1786449 DEBUG (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:37263_solr"} current state version: 0
   [junit4]   2> 1786449 INFO  (zkCallback-6261-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 1786449 INFO  (zkCallback-6247-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 1786449 INFO  (zkCallback-6254-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 1786449 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1786449 INFO  (zkCallback-6269-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 1786449 INFO  (zkCallback-6240-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 1786449 INFO  (coreCloseExecutor-9516-thread-1) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.SolrCore [tlog_replica_test_recovery_shard1_replica_t2]  CLOSING SolrCore org.apache.solr.core.SolrCore@2aacb161
   [junit4]   2> 1786449 INFO  (coreCloseExecutor-9516-thread-1) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_recovery.shard1.replica_t2, tag=2aacb161
   [junit4]   2> 1786449 INFO  (coreCloseExecutor-9516-thread-1) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3511f0f4: rootName = solr_37263, domain = solr.core.tlog_replica_test_recovery.shard1.replica_t2, service url = null, agent id = null] for registry solr.core.tlog_replica_test_recovery.shard1.replica_t2 / com.codahale.metrics.MetricRegistry@7b16d754
   [junit4]   2> 1786455 INFO  (coreCloseExecutor-9516-thread-1) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.tlog_replica_test_recovery.shard1.leader, tag=2aacb161
   [junit4]   2> 1786456 INFO  (coreCloseExecutor-9516-thread-1) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
   [junit4]   2> 1786456 INFO  (coreCloseExecutor-9516-thread-1) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@71eebc24 commitCommandVersion:0
   [junit4]   2> 1786457 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.c.Overseer Overseer (id=72164126028136452-127.0.0.1:37263_solr-n_0000000000) closing
   [junit4]   2> 1786457 INFO  (OverseerStateUpdate-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [n:127.0.0.1:37263_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37263_solr
   [junit4]   2> 1786457 WARN  (OverseerAutoScalingTriggerThread-72164126028136452-127.0.0.1:37263_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1786459 INFO  (zkCallback-6254-thread-1) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34977_solr
   [junit4]   2> 1786459 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2aea7ff8{/solr,null,UNAVAILABLE}
   [junit4]   2> 1786460 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 1786460 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.c.ChaosMonkey monkey: stop jetty! 37263
   [junit4]   2> 1786460 INFO  (zkCallback-6254-thread-1) [n:127.0.0.1:34977_solr    ] o.a.s.c.Overseer Overseer (id=72164126028136455-127.0.0.1:34977_solr-n_0000000001) starting
   [junit4]   2> 1786462 INFO  (qtp254001560-24546) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tlog_replica_test_recovery/terms/shard1 to Terms{values={core_node3=2, core_node4=1}, version=3}
   [junit4]   2> 1786462 INFO  (qtp254001560-24546) [n:127.0.0.1:34977_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[6 (1617167699928940544)]} 0 0
   [junit4]   2> 1786462 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.c.ChaosMonkey monkey: starting jetty! 37263
   [junit4]   2> 1786463 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 9.0.4+11
   [junit4]   2> 1786463 INFO  (OverseerStateUpdate-72164126028136455-127.0.0.1:34977_solr-n_0000000001) [n:127.0.0.1:34977_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:34977_solr
   [junit4]   2> 1786464 DEBUG (OverseerStateUpdate-72164126028136455-127.0.0.1:34977_solr-n_0000000001) [n:127.0.0.1:34977_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:37263_solr"} current state version: 0
   [junit4]   2> 1786465 INFO  (zkCallback-6254-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1786466 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1786466 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1786466 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1786466 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2c9477c{/solr,null,AVAILABLE}
   [junit4]   2> 1786466 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1fb899cd{SSL,[ssl, http/1.1]}{127.0.0.1:37263}
   [junit4]   2> 1786466 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.e.j.s.Server Started @1786557ms
   [junit4]   2> 1786466 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37263}
   [junit4]   2> 1786467 ERROR (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1786467 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1786467 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 1786467 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1786467 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1786467 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-15T03:08:20.750224Z
   [junit4]   2> 1786469 DEBUG (OverseerAutoScalingTriggerThread-72164126028136455-127.0.0.1:34977_solr-n_0000000001) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:34977_solr]
   [junit4]   2> 1786470 DEBUG (OverseerAutoScalingTriggerThread-72164126028136455-127.0.0.1:34977_solr-n_0000000001) [    ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:37263_solr
   [junit4]   2> 1786474 INFO  (zkConnectionManagerCallback-6272-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1786475 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1786477 DEBUG (ScheduledTrigger-9519-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 1786477 DEBUG (ScheduledTrigger-9519-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:37263_solr
   [junit4]   2> 1786615 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35163/solr
   [junit4]   2> 1786616 INFO  (zkConnectionManagerCallback-6276-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1786617 INFO  (zkConnectionManagerCallback-6278-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1786620 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [n:127.0.0.1:37263_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1786622 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [n:127.0.0.1:37263_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1786622 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [n:127.0.0.1:37263_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37263_solr
   [junit4]   2> 1786622 DEBUG (OverseerStateUpdate-72164126028136455-127.0.0.1:34977_solr-n_0000000001) [n:127.0.0.1:34977_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:37263_solr"} current state version: 0
   [junit4]   2> 1786623 INFO  (zkCallback-6261-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1786623 INFO  (zkCallback-6269-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1786623 INFO  (zkCallback-6254-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1786623 INFO  (zkCallback-6277-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1786636 INFO  (zkConnectionManagerCallback-6285-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1786637 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [n:127.0.0.1:37263_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1786637 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [n:127.0.0.1:37263_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35163/solr ready
   [junit4]   2> 1786638 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [n:127.0.0.1:37263_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1786652 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [n:127.0.0.1:37263_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37263.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@613c67f3
   [junit4]   2> 1786662 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [n:127.0.0.1:37263_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37263.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@613c67f3
   [junit4]   2> 1786662 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [n:127.0.0.1:37263_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37263.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@613c67f3
   [junit4]   2> 1786664 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [n:127.0.0.1:37263_solr    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestTlogReplica_EF1D694F57FF56D4-001/tempDir-001/node2/.
   [junit4]   2> 1786664 INFO  (TEST-TestTlogReplica.testRecovery-seed#[EF1D694F57FF56D4]) [n:127.0.0.1:37263_solr    ] o.a.s.c.CorePropertiesLocator Cores are: [tlog_replica_test_recovery_shard1_replica_t2]
   [junit4]   2> 1786669 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 1786673 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr    ] o.a.s.s.IndexSchema [tlog_replica_test_recovery_shard1_replica_t2] Schema name=minimal
   [junit4]   2> 1786675 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1786679 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 1786682 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.s.IndexSchema [tlog_replica_test_recovery_shard1_replica_t2] Schema name=minimal
   [junit4]   2> 1786683 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1786683 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_recovery_shard1_replica_t2' using configuration from collection tlog_replica_test_recovery, trusted=true
   [junit4]   2> 1786683 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37263.solr.core.tlog_replica_test_recovery.shard1.replica_t2' (registry 'solr.core.tlog_replica_test_recovery.shard1.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@613c67f3
   [junit4]   2> 1786684 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.SolrCore [[tlog_replica_test_recovery_shard1_replica_t2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestTlogReplica_EF1D694F57FF56D4-001/tempDir-001/node2/tlog_replica_test_recovery_shard1_replica_t2], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestTlogReplica_EF1D694F57FF56D4-001/tempDir-001/node2/./tlog_replica_test_recovery_shard1_replica_t2/data/]
   [junit4]   2> 1786723 INFO  (zkCallback-6254-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1786741 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1786741 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1786742 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1786742 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1786743 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@34d2baea[tlog_replica_test_recovery_shard1_replica_t2] main]
   [junit4]   2> 1786744 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1786744 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1786744 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0.0.1:37263_solr) [n:127.0.0.1:37263_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1786744 INFO  (coreLoadExecutor-9530-thread-1-processing-n:127.0

[...truncated too long message...]

uster, tag=null
   [junit4]   2> 187849 INFO  (zkCallback-237-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 187849 INFO  (zkCallback-254-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 187849 INFO  (zkCallback-244-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 187850 INFO  (jetty-closer-134-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 187850 INFO  (zkCallback-261-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 187852 INFO  (jetty-closer-134-thread-1) [    ] o.a.s.c.Overseer Overseer (id=72164234059579417-127.0.0.1:40353_solr-n_0000000005) closing
   [junit4]   2> 187852 INFO  (OverseerStateUpdate-72164234059579417-127.0.0.1:40353_solr-n_0000000005) [n:127.0.0.1:40353_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:40353_solr
   [junit4]   2> 187852 WARN  (OverseerAutoScalingTriggerThread-72164234059579417-127.0.0.1:40353_solr-n_0000000005) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 187853 INFO  (jetty-closer-134-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1205562a{/solr,null,UNAVAILABLE}
   [junit4]   2> 187853 INFO  (jetty-closer-134-thread-2) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 187854 INFO  (jetty-closer-134-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2ee5db34{/solr,null,UNAVAILABLE}
   [junit4]   2> 187854 INFO  (jetty-closer-134-thread-1) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 187854 ERROR (SUITE-TestTlogReplica-seed#[EF1D694F57FF56D4]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 187854 INFO  (SUITE-TestTlogReplica-seed#[EF1D694F57FF56D4]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40653 40653
   [junit4]   2> 187935 INFO  (Thread-98) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40653 40653
   [junit4]   2> 187936 WARN  (Thread-98) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	31	/solr/configs/conf
   [junit4]   2> 	31	/solr/collections/tlog_replica_test_create_delete/terms/shard2
   [junit4]   2> 	31	/solr/collections/tlog_replica_test_create_delete/terms/shard1
   [junit4]   2> 	19	/solr/collections/tlog_replica_test_recovery/terms/shard1
   [junit4]   2> 	18	/solr/collections/tlog_replica_test_remove_leader/terms/shard1
   [junit4]   2> 	17	/solr/collections/tlog_replica_test_kill_tlog_replica/terms/shard1
   [junit4]   2> 	16	/solr/collections/tlog_replica_test_basic_leader_election/terms/shard1
   [junit4]   2> 	16	/solr/collections/tlog_replica_test_kill_leader/terms/shard1
   [junit4]   2> 	15	/solr/aliases.json
   [junit4]   2> 	13	/solr/collections/tlog_replica_test_out_of_order_db_qwith_in_place_updates/terms/shard1
   [junit4]   2> 	11	/solr/collections/tlog_replica_test_only_leader_indexes/terms/shard1
   [junit4]   2> 	10	/solr/collections/tlog_replica_test_delete_by_id/terms/shard1
   [junit4]   2> 	10	/solr/collections/tlog_replica_test_real_time_get/terms/shard1
   [junit4]   2> 	8	/solr/collections/tlog_replica_test_add_remove_tlog_replica/terms/shard2
   [junit4]   2> 	7	/solr/security.json
   [junit4]   2> 	7	/solr/collections/tlog_replica_test_add_remove_tlog_replica/terms/shard1
   [junit4]   2> 	4	/solr/collections/tlog_replica_test_add_docs/terms/shard1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	25	/solr/collections/tlog_replica_test_out_of_order_db_qwith_in_place_updates/state.json
   [junit4]   2> 	23	/solr/collections/tlog_replica_test_create_delete/state.json
   [junit4]   2> 	15	/solr/clusterprops.json
   [junit4]   2> 	15	/solr/clusterstate.json
   [junit4]   2> 	14	/solr/collections/tlog_replica_test_kill_leader/state.json
   [junit4]   2> 	13	/solr/collections/tlog_replica_test_basic_leader_election/state.json
   [junit4]   2> 	13	/solr/collections/tlog_replica_test_kill_tlog_replica/state.json
   [junit4]   2> 	9	/solr/collections/tlog_replica_test_remove_leader/state.json
   [junit4]   2> 	8	/solr/collections/tlog_replica_test_recovery/state.json
   [junit4]   2> 	4	/solr/autoscaling.json
   [junit4]   2> 	3	/solr/collections/tlog_replica_test_add_remove_tlog_replica/state.json
   [junit4]   2> 	3	/solr/collections/tlog_replica_test_real_time_get/state.json
   [junit4]   2> 	3	/solr/collections/tlog_replica_test_only_leader_indexes/state.json
   [junit4]   2> 	3	/solr/collections/tlog_replica_test_delete_by_id/state.json
   [junit4]   2> 	2	/solr/overseer_elect/election/72164234059579409-127.0.0.1:34709_solr-n_0000000003
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_create_delete/leader_elect/shard1/election/72164234059579409-core_node9-n_0000000001
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_add_docs/state.json
   [junit4]   2> 	2	/solr/overseer_elect/election/72164234059579398-127.0.0.1:34709_solr-n_0000000000
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/72164234059579409-core_node16-n_0000000000
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_kill_tlog_replica/leader_elect/shard1/election/72164234059579417-core_node4-n_0000000000
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_create_delete/leader_elect/shard1/election/72164234059579413-core_node6-n_0000000000
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_recovery/leader_elect/shard1/election/72164234059579398-core_node3-n_0000000000
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/72164234059579413-core_node15-n_0000000001
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/72164234059579409-core_node11-n_0000000000
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/72164234059579413-core_node13-n_0000000002
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	15	/solr/live_nodes
   [junit4]   2> 	15	/solr/collections
   [junit4]   2> 	4	/solr/overseer/queue
   [junit4]   2> 	4	/solr/autoscaling/events/.scheduled_maintenance
   [junit4]   2> 	4	/solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 	4	/solr/overseer/collection-queue-work
   [junit4]   2> 	4	/solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestTlogReplica_EF1D694F57FF56D4-002
   [junit4]   2> Nov 15, 2018 3:37:11 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 5 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {foo=PostingsFormat(name=Direct), title_s=PostingsFormat(name=Memory), id=PostingsFormat(name=MockRandom), foo_s=PostingsFormat(name=Direct)}, docValues:{_version_=DocValuesFormat(name=Direct), id=DocValuesFormat(name=Lucene70), inplace_updatable_int=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=263, maxMBSortInHeap=5.781191937299397, sim=RandomSimilarity(queryNorm=false): {}, locale=is-IS, timezone=Asia/Bahrain
   [junit4]   2> NOTE: Linux 4.15.0-38-generic amd64/Oracle Corporation 9.0.4 (64-bit)/cpus=8,threads=1,free=7105480,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestTlogReplica, TestTlogReplica]
   [junit4] Completed [5/5 (5!)] on J2 in 87.64s, 13 tests, 1 failure <<< FAILURES!

[...truncated 15 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/common-build.xml:1567: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/common-build.xml:1092: There were test failures: 5 suites, 65 tests, 5 failures [seed: EF1D694F57FF56D4]

Total time: 3 minutes 11 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   5/5 failed: org.apache.solr.cloud.TestTlogReplica
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=c4961d48dfb66223977ce6678f6de94f93f841c9, workspace=/var/lib/jenkins/workspace/Lucene-Solr-7.x-Linux
[WARNINGS] Computing warning deltas based on reference build #3093
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-10.0.1) - Build # 3093 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3093/
Java: 64bit/jdk-10.0.1 -XX:+UseCompressedOops -XX:+UseG1GC

7 tests failed.
FAILED:  org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica

Error Message:
Expected new active leader null Live Nodes: [127.0.0.1:33453_solr, 127.0.0.1:40051_solr, 127.0.0.1:42603_solr] Last available state: DocCollection(raceDeleteReplica_false//collections/raceDeleteReplica_false/state.json/12)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node4":{           "core":"raceDeleteReplica_false_shard1_replica_n3",           "base_url":"https://127.0.0.1:45385/solr",           "node_name":"127.0.0.1:45385_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node6":{           "core":"raceDeleteReplica_false_shard1_replica_n5",           "base_url":"https://127.0.0.1:45385/solr",           "node_name":"127.0.0.1:45385_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected new active leader
null
Live Nodes: [127.0.0.1:33453_solr, 127.0.0.1:40051_solr, 127.0.0.1:42603_solr]
Last available state: DocCollection(raceDeleteReplica_false//collections/raceDeleteReplica_false/state.json/12)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"raceDeleteReplica_false_shard1_replica_n3",
          "base_url":"https://127.0.0.1:45385/solr",
          "node_name":"127.0.0.1:45385_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node6":{
          "core":"raceDeleteReplica_false_shard1_replica_n5",
          "base_url":"https://127.0.0.1:45385/solr",
          "node_name":"127.0.0.1:45385_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
	at __randomizedtesting.SeedInfo.seed([A5E10D5AC7D21626:CFF76C8AAF205CEC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
	at org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:334)
	at org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:230)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest

Error Message:
Error from server at https://127.0.0.1:42603/solr: Cannot create collection delLiveColl. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 3. This allows a maximum of 3 to be created. Value of numShards is 2, value of nrtReplicas is 2, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 4 shards to be created (higher than the allowed number)

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:42603/solr: Cannot create collection delLiveColl. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 3. This allows a maximum of 3 to be created. Value of numShards is 2, value of nrtReplicas is 2, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 4 shards to be created (higher than the allowed number)
	at __randomizedtesting.SeedInfo.seed([A5E10D5AC7D21626:881B951DAEDBE53]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:75)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:40539/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:40539/solr
	at __randomizedtesting.SeedInfo.seed([A5E10D5AC7D21626:641174F6EA82DC81]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
	at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:36915/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:36915/solr
	at __randomizedtesting.SeedInfo.seed([A5E10D5AC7D21626:641174F6EA82DC81]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
	at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:45939/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:45939/solr
	at __randomizedtesting.SeedInfo.seed([A5E10D5AC7D21626:641174F6EA82DC81]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
	at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:43199/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:43199/solr
	at __randomizedtesting.SeedInfo.seed([A5E10D5AC7D21626:641174F6EA82DC81]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
	at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:42413/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:42413/solr
	at __randomizedtesting.SeedInfo.seed([A5E10D5AC7D21626:641174F6EA82DC81]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
	at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 13415 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_A5E10D5AC7D21626-001/init-core-data-001
   [junit4]   2> 762595 WARN  (SUITE-DeleteReplicaTest-seed#[A5E10D5AC7D21626]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=26 numCloses=26
   [junit4]   2> 762595 INFO  (SUITE-DeleteReplicaTest-seed#[A5E10D5AC7D21626]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 762596 INFO  (SUITE-DeleteReplicaTest-seed#[A5E10D5AC7D21626]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
   [junit4]   2> 762596 INFO  (SUITE-DeleteReplicaTest-seed#[A5E10D5AC7D21626]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 762596 INFO  (SUITE-DeleteReplicaTest-seed#[A5E10D5AC7D21626]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_A5E10D5AC7D21626-001/tempDir-001
   [junit4]   2> 762596 INFO  (SUITE-DeleteReplicaTest-seed#[A5E10D5AC7D21626]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 762597 INFO  (Thread-2096) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 762597 INFO  (Thread-2096) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 762602 ERROR (Thread-2096) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 762697 INFO  (SUITE-DeleteReplicaTest-seed#[A5E10D5AC7D21626]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:42331
   [junit4]   2> 762698 INFO  (zkConnectionManagerCallback-2461-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762701 INFO  (jetty-launcher-2458-thread-3) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 10.0.1+10
   [junit4]   2> 762701 INFO  (jetty-launcher-2458-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 10.0.1+10
   [junit4]   2> 762701 INFO  (jetty-launcher-2458-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 10.0.1+10
   [junit4]   2> 762702 INFO  (jetty-launcher-2458-thread-4) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 10.0.1+10
   [junit4]   2> 762704 INFO  (jetty-launcher-2458-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 762704 INFO  (jetty-launcher-2458-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 762704 INFO  (jetty-launcher-2458-thread-4) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 762704 INFO  (jetty-launcher-2458-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 762704 INFO  (jetty-launcher-2458-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 762704 INFO  (jetty-launcher-2458-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 762704 INFO  (jetty-launcher-2458-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1dd02337{/solr,null,AVAILABLE}
   [junit4]   2> 762705 INFO  (jetty-launcher-2458-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@474c58{/solr,null,AVAILABLE}
   [junit4]   2> 762705 INFO  (jetty-launcher-2458-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 762705 INFO  (jetty-launcher-2458-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 762705 INFO  (jetty-launcher-2458-thread-3) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 762705 INFO  (jetty-launcher-2458-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 762705 INFO  (jetty-launcher-2458-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 762705 INFO  (jetty-launcher-2458-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 762706 INFO  (jetty-launcher-2458-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@596024c8{SSL,[ssl, http/1.1]}{127.0.0.1:45385}
   [junit4]   2> 762706 INFO  (jetty-launcher-2458-thread-2) [    ] o.e.j.s.Server Started @762782ms
   [junit4]   2> 762706 INFO  (jetty-launcher-2458-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45385}
   [junit4]   2> 762706 ERROR (jetty-launcher-2458-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 762706 INFO  (jetty-launcher-2458-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 762706 INFO  (jetty-launcher-2458-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 762706 INFO  (jetty-launcher-2458-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 762706 INFO  (jetty-launcher-2458-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 762706 INFO  (jetty-launcher-2458-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-14T22:51:57.581413Z
   [junit4]   2> 762706 INFO  (jetty-launcher-2458-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@32018d22{/solr,null,AVAILABLE}
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@21e0ef44{SSL,[ssl, http/1.1]}{127.0.0.1:40051}
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-4) [    ] o.e.j.s.Server Started @762783ms
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@146c676{/solr,null,AVAILABLE}
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40051}
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@d272560{SSL,[ssl, http/1.1]}{127.0.0.1:42603}
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-3) [    ] o.e.j.s.Server Started @762783ms
   [junit4]   2> 762707 INFO  (zkConnectionManagerCallback-2463-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42603}
   [junit4]   2> 762707 ERROR (jetty-launcher-2458-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-4) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 762707 ERROR (jetty-launcher-2458-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@39851d02{SSL,[ssl, http/1.1]}{127.0.0.1:33453}
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-14T22:51:57.582630Z
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-1) [    ] o.e.j.s.Server Started @762783ms
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33453}
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-14T22:51:57.582751Z
   [junit4]   2> 762707 ERROR (jetty-launcher-2458-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 762707 INFO  (jetty-launcher-2458-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-14T22:51:57.582917Z
   [junit4]   2> 762708 INFO  (zkConnectionManagerCallback-2465-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762708 INFO  (zkConnectionManagerCallback-2467-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762708 INFO  (jetty-launcher-2458-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 762708 INFO  (jetty-launcher-2458-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 762708 INFO  (zkConnectionManagerCallback-2469-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762709 INFO  (jetty-launcher-2458-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 762818 INFO  (jetty-launcher-2458-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42331/solr
   [junit4]   2> 762828 INFO  (zkConnectionManagerCallback-2473-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762830 INFO  (jetty-launcher-2458-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42331/solr
   [junit4]   2> 762838 INFO  (zkConnectionManagerCallback-2476-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762838 INFO  (zkConnectionManagerCallback-2479-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762846 INFO  (jetty-launcher-2458-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42331/solr
   [junit4]   2> 762849 INFO  (zkConnectionManagerCallback-2482-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762854 INFO  (jetty-launcher-2458-thread-2) [n:127.0.0.1:45385_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:45385_solr
   [junit4]   2> 762854 INFO  (jetty-launcher-2458-thread-2) [n:127.0.0.1:45385_solr    ] o.a.s.c.Overseer Overseer (id=72163118127251464-127.0.0.1:45385_solr-n_0000000000) starting
   [junit4]   2> 762858 INFO  (jetty-launcher-2458-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42331/solr
   [junit4]   2> 762865 INFO  (zkConnectionManagerCallback-2495-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762865 INFO  (zkConnectionManagerCallback-2488-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762866 INFO  (jetty-launcher-2458-thread-2) [n:127.0.0.1:45385_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42331/solr ready
   [junit4]   2> 762869 INFO  (zkConnectionManagerCallback-2498-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762873 INFO  (OverseerStateUpdate-72163118127251464-127.0.0.1:45385_solr-n_0000000000) [n:127.0.0.1:45385_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:45385_solr
   [junit4]   2> 762877 INFO  (jetty-launcher-2458-thread-3) [n:127.0.0.1:42603_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42603_solr
   [junit4]   2> 762889 INFO  (zkCallback-2494-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 762889 INFO  (zkCallback-2481-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 762889 INFO  (zkCallback-2475-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 762898 INFO  (zkConnectionManagerCallback-2500-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762898 INFO  (jetty-launcher-2458-thread-2) [n:127.0.0.1:45385_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 762898 INFO  (jetty-launcher-2458-thread-2) [n:127.0.0.1:45385_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45385_solr
   [junit4]   2> 762898 INFO  (zkCallback-2475-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 762899 INFO  (zkCallback-2494-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 762899 INFO  (zkCallback-2481-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 762901 INFO  (jetty-launcher-2458-thread-1) [n:127.0.0.1:33453_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 762902 INFO  (jetty-launcher-2458-thread-1) [n:127.0.0.1:33453_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 762902 INFO  (jetty-launcher-2458-thread-1) [n:127.0.0.1:33453_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33453_solr
   [junit4]   2> 762905 INFO  (zkCallback-2494-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 762905 INFO  (zkCallback-2481-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 762905 INFO  (zkConnectionManagerCallback-2507-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762906 INFO  (jetty-launcher-2458-thread-3) [n:127.0.0.1:42603_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 762906 INFO  (jetty-launcher-2458-thread-3) [n:127.0.0.1:42603_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42331/solr ready
   [junit4]   2> 762910 INFO  (zkConnectionManagerCallback-2502-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762913 INFO  (jetty-launcher-2458-thread-4) [n:127.0.0.1:40051_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 762914 INFO  (zkCallback-2475-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 762918 INFO  (jetty-launcher-2458-thread-4) [n:127.0.0.1:40051_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 762918 INFO  (jetty-launcher-2458-thread-4) [n:127.0.0.1:40051_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40051_solr
   [junit4]   2> 762919 INFO  (zkCallback-2475-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 762919 INFO  (zkCallback-2494-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 762919 INFO  (zkCallback-2481-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 762921 INFO  (jetty-launcher-2458-thread-2) [n:127.0.0.1:45385_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 762925 INFO  (zkCallback-2499-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (4)
   [junit4]   2> 762931 INFO  (jetty-launcher-2458-thread-2) [n:127.0.0.1:45385_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45385.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 762933 INFO  (jetty-launcher-2458-thread-3) [n:127.0.0.1:42603_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 762937 INFO  (zkCallback-2506-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 762938 INFO  (jetty-launcher-2458-thread-2) [n:127.0.0.1:45385_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45385.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 762938 INFO  (jetty-launcher-2458-thread-2) [n:127.0.0.1:45385_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45385.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 762939 INFO  (jetty-launcher-2458-thread-2) [n:127.0.0.1:45385_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_A5E10D5AC7D21626-001/tempDir-001/node2/.
   [junit4]   2> 762941 INFO  (zkCallback-2501-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 762954 INFO  (zkConnectionManagerCallback-2518-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762954 INFO  (jetty-launcher-2458-thread-1) [n:127.0.0.1:33453_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 762956 INFO  (jetty-launcher-2458-thread-1) [n:127.0.0.1:33453_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42331/solr ready
   [junit4]   2> 762957 INFO  (zkConnectionManagerCallback-2523-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 762958 INFO  (jetty-launcher-2458-thread-4) [n:127.0.0.1:40051_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 762958 INFO  (jetty-launcher-2458-thread-4) [n:127.0.0.1:40051_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42331/solr ready
   [junit4]   2> 762963 INFO  (jetty-launcher-2458-thread-3) [n:127.0.0.1:42603_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42603.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 762965 INFO  (jetty-launcher-2458-thread-1) [n:127.0.0.1:33453_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 762966 INFO  (jetty-launcher-2458-thread-4) [n:127.0.0.1:40051_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 762985 INFO  (jetty-launcher-2458-thread-3) [n:127.0.0.1:42603_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42603.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 762985 INFO  (jetty-launcher-2458-thread-3) [n:127.0.0.1:42603_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42603.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 762986 INFO  (jetty-launcher-2458-thread-3) [n:127.0.0.1:42603_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_A5E10D5AC7D21626-001/tempDir-001/node3/.
   [junit4]   2> 762989 INFO  (jetty-launcher-2458-thread-4) [n:127.0.0.1:40051_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40051.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 762990 INFO  (jetty-launcher-2458-thread-1) [n:127.0.0.1:33453_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33453.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 762995 INFO  (jetty-launcher-2458-thread-4) [n:127.0.0.1:40051_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40051.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 762995 INFO  (jetty-launcher-2458-thread-4) [n:127.0.0.1:40051_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40051.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 762995 INFO  (jetty-launcher-2458-thread-1) [n:127.0.0.1:33453_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33453.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 762995 INFO  (jetty-launcher-2458-thread-1) [n:127.0.0.1:33453_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33453.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 762995 INFO  (jetty-launcher-2458-thread-4) [n:127.0.0.1:40051_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_A5E10D5AC7D21626-001/tempDir-001/node4/.
   [junit4]   2> 762996 INFO  (jetty-launcher-2458-thread-1) [n:127.0.0.1:33453_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_A5E10D5AC7D21626-001/tempDir-001/node1/.
   [junit4]   2> 763050 INFO  (zkConnectionManagerCallback-2527-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 763054 INFO  (zkConnectionManagerCallback-2532-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 763055 INFO  (SUITE-DeleteReplicaTest-seed#[A5E10D5AC7D21626]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 763055 INFO  (SUITE-DeleteReplicaTest-seed#[A5E10D5AC7D21626]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42331/solr ready
   [junit4]   2> 763070 INFO  (TEST-DeleteReplicaTest.deleteReplicaByCount-seed#[A5E10D5AC7D21626]) [    ] o.a.s.SolrTestCaseJ4 ###Starting deleteReplicaByCount
   [junit4]   2> 763080 INFO  (qtp876088097-9166) [n:127.0.0.1:45385_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=1&collection.configName=conf&name=deleteByCount&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 763081 INFO  (OverseerThreadFactory-4063-thread-1-processing-n:127.0.0.1:45385_solr) [n:127.0.0.1:45385_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection deleteByCount
   [junit4]   2> 763187 INFO  (qtp876088097-9171) [n:127.0.0.1:45385_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 763197 INFO  (qtp1896482363-9176) [n:127.0.0.1:42603_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 763207 INFO  (qtp837923232-9177) [n:127.0.0.1:33453_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 763218 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 763220 INFO  (OverseerStateUpdate-72163118127251464-127.0.0.1:45385_solr-n_0000000000) [n:127.0.0.1:45385_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteByCount",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"deleteByCount_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:40051/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 763221 INFO  (OverseerStateUpdate-72163118127251464-127.0.0.1:45385_solr-n_0000000000) [n:127.0.0.1:45385_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteByCount",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"deleteByCount_shard1_replica_t2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:33453/solr",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 763221 INFO  (OverseerStateUpdate-72163118127251464-127.0.0.1:45385_solr-n_0000000000) [n:127.0.0.1:45385_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteByCount",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"deleteByCount_shard1_replica_p4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:42603/solr",
   [junit4]   2>   "type":"PULL",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 763426 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr    x:deleteByCount_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=deleteByCount_shard1_replica_n1&action=CREATE&numShards=1&collection=deleteByCount&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 763426 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr    x:deleteByCount_shard1_replica_t2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=deleteByCount_shard1_replica_t2&action=CREATE&numShards=1&collection=deleteByCount&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 763426 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr    x:deleteByCount_shard1_replica_p4] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf&newCollection=true&name=deleteByCount_shard1_replica_p4&action=CREATE&numShards=1&collection=deleteByCount&shard=shard1&wt=javabin&version=2&replicaType=PULL
   [junit4]   2> 763426 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr    x:deleteByCount_shard1_replica_p4] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 764432 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 764432 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 764432 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 764436 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.s.IndexSchema [deleteByCount_shard1_replica_t2] Schema name=minimal
   [junit4]   2> 764436 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.s.IndexSchema [deleteByCount_shard1_replica_p4] Schema name=minimal
   [junit4]   2> 764436 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.s.IndexSchema [deleteByCount_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 764437 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 764437 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 764437 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.c.CoreContainer Creating SolrCore 'deleteByCount_shard1_replica_p4' using configuration from collection deleteByCount, trusted=true
   [junit4]   2> 764437 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 764437 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'deleteByCount_shard1_replica_t2' using configuration from collection deleteByCount, trusted=true
   [junit4]   2> 764437 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'deleteByCount_shard1_replica_n1' using configuration from collection deleteByCount, trusted=true
   [junit4]   2> 764437 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33453.solr.core.deleteByCount.shard1.replica_t2' (registry 'solr.core.deleteByCount.shard1.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 764437 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42603.solr.core.deleteByCount.shard1.replica_p4' (registry 'solr.core.deleteByCount.shard1.replica_p4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 764437 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40051.solr.core.deleteByCount.shard1.replica_n1' (registry 'solr.core.deleteByCount.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5739929b
   [junit4]   2> 764438 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.SolrCore [[deleteByCount_shard1_replica_t2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_A5E10D5AC7D21626-001/tempDir-001/node1/deleteByCount_shard1_replica_t2], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_A5E10D5AC7D21626-001/tempDir-001/node1/./deleteByCount_shard1_replica_t2/data/]
   [junit4]   2> 764438 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.c.SolrCore [[deleteByCount_shard1_replica_p4] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_A5E10D5AC7D21626-001/tempDir-001/node3/deleteByCount_shard1_replica_p4], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_A5E10D5AC7D21626-001/tempDir-001/node3/./deleteByCount_shard1_replica_p4/data/]
   [junit4]   2> 764438 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.c.SolrCore [[deleteByCount_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_A5E10D5AC7D21626-001/tempDir-001/node4/deleteByCount_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_A5E10D5AC7D21626-001/tempDir-001/node4/./deleteByCount_shard1_replica_n1/data/]
   [junit4]   2> 764466 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 764466 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 764466 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 764466 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 764466 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.s.SolrIndexSearcher Opening [Searcher@4d191f3f[deleteByCount_shard1_replica_p4] main]
   [junit4]   2> 764467 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 764467 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 764467 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 764467 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 764468 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@17b9ec9[deleteByCount_shard1_replica_t2] main]
   [junit4]   2> 764468 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 764468 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 764468 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 764469 INFO  (searcherExecutor-4079-thread-1-processing-n:127.0.0.1:42603_solr x:deleteByCount_shard1_replica_p4 c:deleteByCount s:shard1 r:core_node6) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.c.SolrCore [deleteByCount_shard1_replica_p4] Registered new searcher Searcher@4d191f3f[deleteByCount_shard1_replica_p4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 764469 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 764469 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1617151571362054144
   [junit4]   2> 764469 INFO  (searcherExecutor-4080-thread-1-processing-n:127.0.0.1:33453_solr x:deleteByCount_shard1_replica_t2 c:deleteByCount s:shard1 r:core_node5) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.SolrCore [deleteByCount_shard1_replica_t2] Registered new searcher Searcher@17b9ec9[deleteByCount_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 764470 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.c.ZkController deleteByCount_shard1_replica_p4 starting background replication from leader
   [junit4]   2> 764470 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:01
   [junit4]   2> 764470 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 764470 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 764471 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deleteByCount/terms/shard1 to Terms{values={core_node5=0}, version=0}
   [junit4]   2> 764472 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 764479 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 764479 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 764480 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 764480 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 764481 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@795e3fd2[deleteByCount_shard1_replica_n1] main]
   [junit4]   2> 764482 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 764482 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 764483 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 764483 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1617151571376734208
   [junit4]   2> 764483 INFO  (searcherExecutor-4081-thread-1-processing-n:127.0.0.1:40051_solr x:deleteByCount_shard1_replica_n1 c:deleteByCount s:shard1 r:core_node3) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.c.SolrCore [deleteByCount_shard1_replica_n1] Registered new searcher Searcher@795e3fd2[deleteByCount_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 764485 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deleteByCount/terms/shard1 to Terms{values={core_node3=0, core_node5=0}, version=1}
   [junit4]   2> 764974 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 764974 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 764974 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33453/solr/deleteByCount_shard1_replica_t2/
   [junit4]   2> 764974 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.u.PeerSync PeerSync: core=deleteByCount_shard1_replica_t2 url=https://127.0.0.1:33453/solr START replicas=[https://127.0.0.1:40051/solr/deleteByCount_shard1_replica_n1/] nUpdates=100
   [junit4]   2> 764975 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.u.PeerSync PeerSync: core=deleteByCount_shard1_replica_t2 url=https://127.0.0.1:33453/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 764977 INFO  (qtp1117861226-9292) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.c.S.Request [deleteByCount_shard1_replica_n1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 764977 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 764977 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 764977 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 764978 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.ZkController deleteByCount_shard1_replica_t2 stopping background replication from leader
   [junit4]   2> 764981 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:33453/solr/deleteByCount_shard1_replica_t2/ shard1
   [junit4]   2> 765133 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 765164 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=deleteByCount_shard1_replica_t2&action=CREATE&numShards=1&collection=deleteByCount&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1737
   [junit4]   2> 765235 INFO  (zkCallback-2499-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 765430 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.h.IndexFetcher Updated masterUrl to https://127.0.0.1:33453/solr/deleteByCount_shard1_replica_t2/
   [junit4]   2> 765432 INFO  (qtp837923232-9181) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.S.Request [deleteByCount_shard1_replica_t2]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 765433 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 765433 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 765433 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 765433 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 765433 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
   [junit4]   2> 765433 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 765433 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@320fbe7e commitCommandVersion:0
   [junit4]   2> 765434 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@db8215b[deleteByCount_shard1_replica_p4] main]
   [junit4]   2> 765434 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 765435 INFO  (searcherExecutor-4079-thread-1) [    ] o.a.s.c.SolrCore [deleteByCount_shard1_replica_p4] Registered new searcher Searcher@db8215b[deleteByCount_shard1_replica_p4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 765479 INFO  (qtp1896482363-9180) [n:127.0.0.1:42603_solr c:deleteByCount s:shard1 r:core_node6 x:deleteByCount_shard1_replica_p4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf&newCollection=true&name=deleteByCount_shard1_replica_p4&action=CREATE&numShards=1&collection=deleteByCount&shard=shard1&wt=javabin&version=2&replicaType=PULL} status=0 QTime=2053
   [junit4]   2> 765490 INFO  (qtp1117861226-9163) [n:127.0.0.1:40051_solr c:deleteByCount s:shard1 r:core_node3 x:deleteByCount_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=deleteByCount_shard1_replica_n1&action=CREATE&numShards=1&collection=deleteByCount&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2064
   [junit4]   2> 765491 INFO  (qtp876088097-9166) [n:127.0.0.1:45385_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 765590 INFO  (zkCallback-2501-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 765590 INFO  (zkCallback-2499-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 765590 INFO  (zkCallback-2475-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCount/state.json] for collection [deleteByCount] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 766429 INFO  (qtp837923232-9293) [n:127.0.0.1:33453_solr c:deleteByCount s:shard1 r:core_node5 x:deleteByCount_shard1_replica_t2] o.a.s.c.S.Request [deleteByCount_shard1_replica_t2]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 766430 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 766430 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 766430 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.h.IndexFetcher Slave's generation: 2
   [junit4]   2> 766430 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.h.IndexFetcher Slave's version: 1542235920308
   [junit4]   2> 766430 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
   [junit4]   2> 766430 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 766430 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@320fbe7e commitCommandVersion:0
   [junit4]   2> 766430 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@5fe312cb[deleteByCount_shard1_replica_p4] main]
   [junit4]   2> 766430 INFO  (indexFetcher-4091-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 766431 INFO  (searcherExecutor-4079-thread-1) [    ] o.a.s.c.SolrCore [deleteByCount_shard1_replica_p4] Registered new searcher Searcher@5fe312cb[deleteByCount_shard1_replica_p4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 766493 INFO  (qtp876088097-9166) [n:127.0.0.1:45385_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=1&collection.configName=conf&name=deleteByCount&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=1&wt=javabin&version=2} status=0 QTime=3412
   [junit4]   2> 766494 INFO  (qtp876088097-9168) [n:127.0.0.1:45385_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :deletereplica with params count=2&action=DELETEREPLICA&collection=deleteByCount&shard=shard1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 766497 INFO  (OverseerCollectionConfigSetProcessor-72163118127251464-127.0.0.1:45385_solr-n_0000000000) [n:127.0.0.1:45385_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 766498 INFO  (qtp1896482363-9176) [n:127.0.0.1:42603_solr    x:deleteByCount_shard1_replica_p4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteByCount.shard1.replica_p4, tag=null
   [junit4]   2> 766498 INFO  (qtp1896482363-9176) [n:127.0.0.1:42603_solr    x:deleteByCount_shard1_replica_p4] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7dc2c29f: rootName = solr_42603, domain = solr.core.deleteByCount.shard1.replica_p4, service url = null, agent id = null] for registry solr.core.deleteByCount.shard1.replica_p4 / com.codahale.metrics.MetricRegistry@48675e15
   [junit4]   2> 766505 INFO  (qtp1896482363-9176) [n:127.0.0.1:42603_solr    x:deleteByCount_shard1_replica_p4] o.a.s.c.ZkController deleteByCount_shard1_replica_p4 stopping background replication from leader
   [junit4]   2> 766506 INFO  (qtp1896482363-9176) [n:127.0.0.1:42603_solr    x:deleteByCount_shard1_replica_p4] o.a.s.c.SolrCore [deleteByCount_shard1_replica_p4]  CLOSING SolrCore org.apache.solr.core.SolrCore@946b2a2
   [junit4]   2> 766506 INFO  (qtp1896482363-9176) [n:127.0.0.1:42603_solr    x:deleteByCount_shard1_replica_p4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteByCount.shard1.replica_p4, tag=946b2a2
   [junit4]   2> 766506 INFO  (qtp1896482363-9176) [n:127.0.0.1:42603_solr    x:deleteByCount_shard1_replica_p4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.deleteByCount.shard1.leader, tag=946b2a2

[...truncated too long message...]

e nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 201471 INFO  (zkCallback-261-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 201471 INFO  (jetty-closer-144-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 201471 INFO  (zkCallback-268-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 201471 INFO  (jetty-closer-144-thread-2) [    ] o.a.s.c.Overseer Overseer (id=72163296347619353-127.0.0.1:43197_solr-n_0000000005) closing
   [junit4]   2> 201471 INFO  (zkCallback-251-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 201472 INFO  (OverseerStateUpdate-72163296347619353-127.0.0.1:43197_solr-n_0000000005) [n:127.0.0.1:43197_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:43197_solr
   [junit4]   2> 201472 INFO  (zkCallback-261-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 201472 INFO  (zkCallback-244-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 201474 WARN  (OverseerAutoScalingTriggerThread-72163296347619353-127.0.0.1:43197_solr-n_0000000005) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 201475 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006022bc9f001a, likely client has closed socket
   [junit4]   2> 201475 INFO  (jetty-closer-144-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@63e2835d{/solr,null,UNAVAILABLE}
   [junit4]   2> 201475 INFO  (jetty-closer-144-thread-1) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 201476 INFO  (jetty-closer-144-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@8c3ddc1{/solr,null,UNAVAILABLE}
   [junit4]   2> 201476 INFO  (jetty-closer-144-thread-2) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 201480 ERROR (SUITE-TestTlogReplica-seed#[A5E10D5AC7D21626]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 201480 INFO  (SUITE-TestTlogReplica-seed#[A5E10D5AC7D21626]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42955 42955
   [junit4]   2> 201512 INFO  (Thread-288) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42955 42955
   [junit4]   2> 201513 WARN  (Thread-288) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	32	/solr/collections/tlog_replica_test_create_delete/terms/shard1
   [junit4]   2> 	31	/solr/configs/conf
   [junit4]   2> 	31	/solr/collections/tlog_replica_test_create_delete/terms/shard2
   [junit4]   2> 	19	/solr/collections/tlog_replica_test_recovery/terms/shard1
   [junit4]   2> 	19	/solr/collections/tlog_replica_test_remove_leader/terms/shard1
   [junit4]   2> 	17	/solr/collections/tlog_replica_test_basic_leader_election/terms/shard1
   [junit4]   2> 	16	/solr/collections/tlog_replica_test_kill_leader/terms/shard1
   [junit4]   2> 	16	/solr/collections/tlog_replica_test_kill_tlog_replica/terms/shard1
   [junit4]   2> 	15	/solr/aliases.json
   [junit4]   2> 	13	/solr/collections/tlog_replica_test_out_of_order_db_qwith_in_place_updates/terms/shard1
   [junit4]   2> 	12	/solr/collections/tlog_replica_test_real_time_get/terms/shard1
   [junit4]   2> 	11	/solr/collections/tlog_replica_test_delete_by_id/terms/shard1
   [junit4]   2> 	11	/solr/collections/tlog_replica_test_only_leader_indexes/terms/shard1
   [junit4]   2> 	8	/solr/collections/tlog_replica_test_add_remove_tlog_replica/terms/shard2
   [junit4]   2> 	7	/solr/security.json
   [junit4]   2> 	7	/solr/collections/tlog_replica_test_add_remove_tlog_replica/terms/shard1
   [junit4]   2> 	4	/solr/collections/tlog_replica_test_add_docs/terms/shard1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	28	/solr/collections/tlog_replica_test_out_of_order_db_qwith_in_place_updates/state.json
   [junit4]   2> 	26	/solr/collections/tlog_replica_test_create_delete/state.json
   [junit4]   2> 	15	/solr/clusterprops.json
   [junit4]   2> 	15	/solr/clusterstate.json
   [junit4]   2> 	13	/solr/collections/tlog_replica_test_recovery/state.json
   [junit4]   2> 	11	/solr/collections/tlog_replica_test_basic_leader_election/state.json
   [junit4]   2> 	9	/solr/collections/tlog_replica_test_remove_leader/state.json
   [junit4]   2> 	8	/solr/collections/tlog_replica_test_kill_leader/state.json
   [junit4]   2> 	8	/solr/collections/tlog_replica_test_kill_tlog_replica/state.json
   [junit4]   2> 	5	/solr/collections/tlog_replica_test_real_time_get/state.json
   [junit4]   2> 	4	/solr/autoscaling.json
   [junit4]   2> 	3	/solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/72163296347619332-core_node11-n_0000000000
   [junit4]   2> 	3	/solr/collections/tlog_replica_test_add_remove_tlog_replica/state.json
   [junit4]   2> 	3	/solr/collections/tlog_replica_test_only_leader_indexes/state.json
   [junit4]   2> 	3	/solr/collections/tlog_replica_test_delete_by_id/state.json
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_recovery/leader_elect/shard1/election/72163296347619341-core_node4-n_0000000000
   [junit4]   2> 	2	/solr/overseer_elect/election/72163296347619341-127.0.0.1:43197_solr-n_0000000002
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_add_docs/state.json
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/72163296347619332-core_node16-n_0000000002
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_create_delete/leader_elect/shard1/election/72163296347619332-core_node9-n_0000000000
   [junit4]   2> 	2	/solr/overseer_elect/election/72163296347619332-127.0.0.1:45939_solr-n_0000000000
   [junit4]   2> 	2	/solr/collections/tlog_replica_test_create_delete/leader_elect/shard1/election/72163296347619341-core_node7-n_0000000002
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	15	/solr/live_nodes
   [junit4]   2> 	15	/solr/collections
   [junit4]   2> 	4	/solr/overseer/queue
   [junit4]   2> 	4	/solr/autoscaling/events/.scheduled_maintenance
   [junit4]   2> 	4	/solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 	4	/solr/overseer/queue-work
   [junit4]   2> 	4	/solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_A5E10D5AC7D21626-001
   [junit4]   2> Nov 14, 2018 11:38:53 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 6 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {foo=PostingsFormat(name=Memory), title_s=PostingsFormat(name=Asserting), id=PostingsFormat(name=Memory), foo_s=PostingsFormat(name=Memory)}, docValues:{_version_=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Memory), inplace_updatable_int=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=128, maxMBSortInHeap=5.872598316600958, sim=RandomSimilarity(queryNorm=true): {}, locale=de, timezone=America/Guyana
   [junit4]   2> NOTE: Linux 4.15.0-38-generic amd64/Oracle Corporation 10.0.1 (64-bit)/cpus=8,threads=1,free=219975296,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [DeleteReplicaTest, TestTlogReplica]
   [junit4] Completed [6/10 (4!)] on J1 in 96.79s, 13 tests, 1 failure <<< FAILURES!

[...truncated 60 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/common-build.xml:1567: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/common-build.xml:1092: There were test failures: 10 suites, 100 tests, 4 failures [seed: A5E10D5AC7D21626]

Total time: 6 minutes 45 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.DeleteReplicaTest
[repro]   4/5 failed: org.apache.solr.cloud.TestTlogReplica
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=b502ba2882a86958ef8769c3cb2fd65cf2d9c7e1, workspace=/var/lib/jenkins/workspace/Lucene-Solr-7.x-Linux
[WARNINGS] Computing warning deltas based on reference build #3092
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2