You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2018/01/02 14:17:52 UTC

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk1.8.0_144) - Build # 1102 - Still Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1102/
Java: 64bit/jdk1.8.0_144 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:     1) Thread[id=28551, name=jetty-launcher-7667-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)    2) Thread[id=28547, name=jetty-launcher-7667-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   1) Thread[id=28551, name=jetty-launcher-7667-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   2) Thread[id=28547, name=jetty-launcher-7667-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
	at __randomizedtesting.SeedInfo.seed([DDA44F59B76C1CFD]:0)


FAILED:  org.apache.solr.cloud.AsyncCallRequestStatusResponseTest.testAsyncCallStatusResponse

Error Message:
Expected collection 'asynccall' to have 2 shards and 1 replica null Live Nodes: [127.0.0.1:32821_solr, 127.0.0.1:42081_solr] Last available state: DocCollection(asynccall//collections/asynccall/state.json/3)={   "pullReplicas":"0",   "replicationFactor":"1",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{"core_node3":{           "core":"asynccall_shard1_replica_n1",           "base_url":"https://127.0.0.1:32821/solr",           "node_name":"127.0.0.1:32821_solr",           "state":"down",           "type":"NRT",           "leader":"true"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node4":{           "core":"asynccall_shard2_replica_n2",           "base_url":"https://127.0.0.1:42081/solr",           "node_name":"127.0.0.1:42081_solr",           "state":"down",           "type":"NRT",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected collection 'asynccall' to have 2 shards and 1 replica
null
Live Nodes: [127.0.0.1:32821_solr, 127.0.0.1:42081_solr]
Last available state: DocCollection(asynccall//collections/asynccall/state.json/3)={
  "pullReplicas":"0",
  "replicationFactor":"1",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{"core_node3":{
          "core":"asynccall_shard1_replica_n1",
          "base_url":"https://127.0.0.1:32821/solr",
          "node_name":"127.0.0.1:32821_solr",
          "state":"down",
          "type":"NRT",
          "leader":"true"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node4":{
          "core":"asynccall_shard2_replica_n2",
          "base_url":"https://127.0.0.1:42081/solr",
          "node_name":"127.0.0.1:42081_solr",
          "state":"down",
          "type":"NRT",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0"}
	at __randomizedtesting.SeedInfo.seed([DDA44F59B76C1CFD:87A52AB599569124]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:269)
	at org.apache.solr.cloud.AsyncCallRequestStatusResponseTest.testAsyncCallStatusResponse(AsyncCallRequestStatusResponseTest.java:43)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12694 lines...]
   [junit4] Suite: org.apache.solr.cloud.AsyncCallRequestStatusResponseTest
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.AsyncCallRequestStatusResponseTest_DDA44F59B76C1CFD-001/init-core-data-001
   [junit4]   2> 979085 WARN  (SUITE-AsyncCallRequestStatusResponseTest-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=6 numCloses=6
   [junit4]   2> 979085 INFO  (SUITE-AsyncCallRequestStatusResponseTest-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 979086 INFO  (SUITE-AsyncCallRequestStatusResponseTest-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 979087 INFO  (SUITE-AsyncCallRequestStatusResponseTest-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 979088 INFO  (SUITE-AsyncCallRequestStatusResponseTest-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.AsyncCallRequestStatusResponseTest_DDA44F59B76C1CFD-001/tempDir-001
   [junit4]   2> 979088 INFO  (SUITE-AsyncCallRequestStatusResponseTest-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 979088 INFO  (Thread-5642) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 979088 INFO  (Thread-5642) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 979090 ERROR (Thread-5642) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 979188 INFO  (SUITE-AsyncCallRequestStatusResponseTest-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:40263
   [junit4]   2> 979196 INFO  (zkConnectionManagerCallback-5600-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 979198 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x160b71330650000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 979200 INFO  (jetty-launcher-5597-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 979201 INFO  (jetty-launcher-5597-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 979202 INFO  (jetty-launcher-5597-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@669c6103{/solr,null,AVAILABLE}
   [junit4]   2> 979202 INFO  (jetty-launcher-5597-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d01640{/solr,null,AVAILABLE}
   [junit4]   2> 979202 INFO  (jetty-launcher-5597-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4d37a027{SSL,[ssl, http/1.1]}{127.0.0.1:42081}
   [junit4]   2> 979202 INFO  (jetty-launcher-5597-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1c5068b0{SSL,[ssl, http/1.1]}{127.0.0.1:32821}
   [junit4]   2> 979202 INFO  (jetty-launcher-5597-thread-1) [    ] o.e.j.s.Server Started @981101ms
   [junit4]   2> 979202 INFO  (jetty-launcher-5597-thread-2) [    ] o.e.j.s.Server Started @981101ms
   [junit4]   2> 979202 INFO  (jetty-launcher-5597-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42081}
   [junit4]   2> 979202 INFO  (jetty-launcher-5597-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=32821}
   [junit4]   2> 979203 ERROR (jetty-launcher-5597-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 979203 ERROR (jetty-launcher-5597-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 979203 INFO  (jetty-launcher-5597-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 979203 INFO  (jetty-launcher-5597-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 979203 INFO  (jetty-launcher-5597-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 979203 INFO  (jetty-launcher-5597-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 979203 INFO  (jetty-launcher-5597-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 979203 INFO  (jetty-launcher-5597-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 979203 INFO  (jetty-launcher-5597-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-02T13:32:56.407Z
   [junit4]   2> 979203 INFO  (jetty-launcher-5597-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-02T13:32:56.407Z
   [junit4]   2> 979205 INFO  (zkConnectionManagerCallback-5603-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 979205 INFO  (jetty-launcher-5597-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 979232 INFO  (zkConnectionManagerCallback-5604-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 979238 INFO  (jetty-launcher-5597-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 979254 INFO  (jetty-launcher-5597-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40263/solr
   [junit4]   2> 979263 INFO  (zkConnectionManagerCallback-5608-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 979267 INFO  (jetty-launcher-5597-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40263/solr
   [junit4]   2> 979298 INFO  (zkConnectionManagerCallback-5612-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 979301 INFO  (zkConnectionManagerCallback-5614-thread-1-processing-n:127.0.0.1:42081_solr) [n:127.0.0.1:42081_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 979313 INFO  (zkConnectionManagerCallback-5616-thread-1-processing-n:127.0.0.1:32821_solr) [n:127.0.0.1:32821_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 979319 INFO  (jetty-launcher-5597-thread-2) [n:127.0.0.1:32821_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 979320 INFO  (jetty-launcher-5597-thread-2) [n:127.0.0.1:32821_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:32821_solr
   [junit4]   2> 979320 INFO  (jetty-launcher-5597-thread-2) [n:127.0.0.1:32821_solr    ] o.a.s.c.Overseer Overseer (id=99280484846338054-127.0.0.1:32821_solr-n_0000000000) starting
   [junit4]   2> 979335 INFO  (jetty-launcher-5597-thread-1) [n:127.0.0.1:42081_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 979335 INFO  (jetty-launcher-5597-thread-1) [n:127.0.0.1:42081_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42081_solr
   [junit4]   2> 979343 INFO  (zkCallback-5613-thread-1-processing-n:127.0.0.1:42081_solr) [n:127.0.0.1:42081_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 979344 INFO  (OverseerStateUpdate-99280484846338054-127.0.0.1:32821_solr-n_0000000000) [n:127.0.0.1:32821_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 979344 INFO  (jetty-launcher-5597-thread-2) [n:127.0.0.1:32821_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32821_solr
   [junit4]   2> 979345 INFO  (zkCallback-5613-thread-1-processing-n:127.0.0.1:42081_solr) [n:127.0.0.1:42081_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 979345 INFO  (zkCallback-5615-thread-1-processing-n:127.0.0.1:32821_solr) [n:127.0.0.1:32821_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 979381 INFO  (jetty-launcher-5597-thread-1) [n:127.0.0.1:42081_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42081.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@100c4f86
   [junit4]   2> 979386 INFO  (jetty-launcher-5597-thread-1) [n:127.0.0.1:42081_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42081.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@100c4f86
   [junit4]   2> 979386 INFO  (jetty-launcher-5597-thread-1) [n:127.0.0.1:42081_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42081.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@100c4f86
   [junit4]   2> 979387 INFO  (jetty-launcher-5597-thread-1) [n:127.0.0.1:42081_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.AsyncCallRequestStatusResponseTest_DDA44F59B76C1CFD-001/tempDir-001/node1/.
   [junit4]   2> 979460 INFO  (jetty-launcher-5597-thread-2) [n:127.0.0.1:32821_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_32821.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@100c4f86
   [junit4]   2> 979467 INFO  (jetty-launcher-5597-thread-2) [n:127.0.0.1:32821_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_32821.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@100c4f86
   [junit4]   2> 979467 INFO  (jetty-launcher-5597-thread-2) [n:127.0.0.1:32821_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_32821.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@100c4f86
   [junit4]   2> 979468 INFO  (jetty-launcher-5597-thread-2) [n:127.0.0.1:32821_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.AsyncCallRequestStatusResponseTest_DDA44F59B76C1CFD-001/tempDir-001/node2/.
   [junit4]   2> 979483 INFO  (zkConnectionManagerCallback-5622-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 979485 INFO  (zkConnectionManagerCallback-5626-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 979486 INFO  (SUITE-AsyncCallRequestStatusResponseTest-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 979486 INFO  (SUITE-AsyncCallRequestStatusResponseTest-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40263/solr ready
   [junit4]   2> 979495 INFO  (TEST-AsyncCallRequestStatusResponseTest.testAsyncCallStatusResponse-seed#[DDA44F59B76C1CFD]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testAsyncCallStatusResponse
   [junit4]   2> 979507 INFO  (qtp1566498685-18816) [n:127.0.0.1:32821_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params async=caeae54f-a8d3-45f5-916d-881a112a8820&replicationFactor=1&collection.configName=conf&name=asynccall&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 979508 INFO  (qtp1566498685-18816) [n:127.0.0.1:32821_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={async=caeae54f-a8d3-45f5-916d-881a112a8820&replicationFactor=1&collection.configName=conf&name=asynccall&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 979512 INFO  (OverseerThreadFactory-3459-thread-1-processing-n:127.0.0.1:32821_solr) [n:127.0.0.1:32821_solr    ] o.a.s.c.CreateCollectionCmd Create collection asynccall
   [junit4]   2> 979515 INFO  (zkCallback-5625-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/asynccall/state.json] for collection [asynccall] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 979616 INFO  (OverseerStateUpdate-99280484846338054-127.0.0.1:32821_solr-n_0000000000) [n:127.0.0.1:32821_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"asynccall",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"asynccall_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:32821/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 979617 INFO  (OverseerStateUpdate-99280484846338054-127.0.0.1:32821_solr-n_0000000000) [n:127.0.0.1:32821_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"asynccall",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"asynccall_shard2_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:42081/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 979717 INFO  (zkCallback-5625-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/asynccall/state.json] for collection [asynccall] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 979822 INFO  (qtp1566498685-18815) [n:127.0.0.1:32821_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&collection=asynccall&version=2&replicaType=NRT&async=caeae54f-a8d3-45f5-916d-881a112a88202601118792265481&coreNodeName=core_node3&name=asynccall_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=0 QTime=0
   [junit4]   2> 979822 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&collection=asynccall&version=2&replicaType=NRT&async=caeae54f-a8d3-45f5-916d-881a112a88202601118792265481&coreNodeName=core_node3&name=asynccall_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 979822 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 979836 INFO  (qtp2126475454-18808) [n:127.0.0.1:42081_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&collection=asynccall&version=2&replicaType=NRT&async=caeae54f-a8d3-45f5-916d-881a112a88202601118793163291&coreNodeName=core_node4&name=asynccall_shard2_replica_n2&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=0 QTime=0
   [junit4]   2> 979836 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&collection=asynccall&version=2&replicaType=NRT&async=caeae54f-a8d3-45f5-916d-881a112a88202601118793163291&coreNodeName=core_node4&name=asynccall_shard2_replica_n2&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 979836 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 979837 INFO  (qtp2126475454-18810) [n:127.0.0.1:42081_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : caeae54f-a8d3-45f5-916d-881a112a88202601118793163291
   [junit4]   2> 979837 INFO  (qtp2126475454-18810) [n:127.0.0.1:42081_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=caeae54f-a8d3-45f5-916d-881a112a88202601118793163291&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 979939 INFO  (zkCallback-5625-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/asynccall/state.json] for collection [asynccall] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 979939 INFO  (zkCallback-5615-thread-1-processing-n:127.0.0.1:32821_solr) [n:127.0.0.1:32821_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/asynccall/state.json] for collection [asynccall] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 979939 INFO  (zkCallback-5613-thread-1-processing-n:127.0.0.1:42081_solr) [n:127.0.0.1:42081_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/asynccall/state.json] for collection [asynccall] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 979939 INFO  (zkCallback-5613-thread-2-processing-n:127.0.0.1:42081_solr) [n:127.0.0.1:42081_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/asynccall/state.json] for collection [asynccall] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 979939 INFO  (zkCallback-5615-thread-2-processing-n:127.0.0.1:32821_solr) [n:127.0.0.1:32821_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/asynccall/state.json] for collection [asynccall] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 980835 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 980837 INFO  (qtp2126475454-18812) [n:127.0.0.1:42081_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : caeae54f-a8d3-45f5-916d-881a112a88202601118793163291
   [junit4]   2> 980838 INFO  (qtp2126475454-18812) [n:127.0.0.1:42081_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=caeae54f-a8d3-45f5-916d-881a112a88202601118793163291&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 980841 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.s.IndexSchema [asynccall_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 980843 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 980843 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'asynccall_shard1_replica_n1' using configuration from collection asynccall, trusted=true
   [junit4]   2> 980844 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_32821.solr.core.asynccall.shard1.replica_n1' (registry 'solr.core.asynccall.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@100c4f86
   [junit4]   2> 980844 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 980844 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.c.SolrCore [[asynccall_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.AsyncCallRequestStatusResponseTest_DDA44F59B76C1CFD-001/tempDir-001/node2/asynccall_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.AsyncCallRequestStatusResponseTest_DDA44F59B76C1CFD-001/tempDir-001/node2/./asynccall_shard1_replica_n1/data/]
   [junit4]   2> 980850 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 980866 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.s.IndexSchema [asynccall_shard2_replica_n2] Schema name=minimal
   [junit4]   2> 980868 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 980868 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'asynccall_shard2_replica_n2' using configuration from collection asynccall, trusted=true
   [junit4]   2> 980869 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42081.solr.core.asynccall.shard2.replica_n2' (registry 'solr.core.asynccall.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@100c4f86
   [junit4]   2> 980869 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 980869 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.c.SolrCore [[asynccall_shard2_replica_n2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.AsyncCallRequestStatusResponseTest_DDA44F59B76C1CFD-001/tempDir-001/node1/asynccall_shard2_replica_n2], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.AsyncCallRequestStatusResponseTest_DDA44F59B76C1CFD-001/tempDir-001/node1/./asynccall_shard2_replica_n2/data/]
   [junit4]   2> 980889 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 980889 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 980890 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 980890 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 980891 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@40bc66fe[asynccall_shard1_replica_n1] main]
   [junit4]   2> 980892 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 980892 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 980893 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 980893 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588487759433039872
   [junit4]   2> 980894 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 980894 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 980895 INFO  (searcherExecutor-3464-thread-1-processing-x:asynccall_shard1_replica_n1 r:core_node3 caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE n:127.0.0.1:32821_solr s:shard1 c:asynccall) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.c.SolrCore [asynccall_shard1_replica_n1] Registered new searcher Searcher@40bc66fe[asynccall_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 980895 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 980895 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 980895 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 980895 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 980895 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:32821/solr/asynccall_shard1_replica_n1/
   [junit4]   2> 980895 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 980895 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:32821/solr/asynccall_shard1_replica_n1/ has no replicas
   [junit4]   2> 980895 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 980896 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@5c2a7a37[asynccall_shard2_replica_n2] main]
   [junit4]   2> 980897 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:32821/solr/asynccall_shard1_replica_n1/ shard1
   [junit4]   2> 980897 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 980897 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 980897 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 980898 INFO  (searcherExecutor-3465-thread-1-processing-x:asynccall_shard2_replica_n2 r:core_node4 caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE n:127.0.0.1:42081_solr s:shard2 c:asynccall) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.c.SolrCore [asynccall_shard2_replica_n2] Registered new searcher Searcher@5c2a7a37[asynccall_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 980898 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588487759438282752
   [junit4]   2> 980905 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 980905 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 980905 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:42081/solr/asynccall_shard2_replica_n2/
   [junit4]   2> 980905 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 980905 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.c.SyncStrategy https://127.0.0.1:42081/solr/asynccall_shard2_replica_n2/ has no replicas
   [junit4]   2> 980905 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 980906 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:42081/solr/asynccall_shard2_replica_n2/ shard2
   [junit4]   2> 981008 INFO  (zkCallback-5625-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/asynccall/state.json] for collection [asynccall] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 981008 INFO  (zkCallback-5613-thread-1-processing-n:127.0.0.1:42081_solr) [n:127.0.0.1:42081_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/asynccall/state.json] for collection [asynccall] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 981008 INFO  (zkCallback-5615-thread-1-processing-n:127.0.0.1:32821_solr) [n:127.0.0.1:32821_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/asynccall/state.json] for collection [asynccall] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 981008 INFO  (zkCallback-5613-thread-2-processing-n:127.0.0.1:42081_solr) [n:127.0.0.1:42081_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/asynccall/state.json] for collection [asynccall] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 981008 INFO  (zkCallback-5615-thread-2-processing-n:127.0.0.1:32821_solr) [n:127.0.0.1:32821_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/asynccall/state.json] for collection [asynccall] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 981048 INFO  (parallelCoreAdminExecutor-3462-thread-1-processing-n:127.0.0.1:32821_solr caeae54f-a8d3-45f5-916d-881a112a88202601118792265481 CREATE) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 981058 INFO  (parallelCoreAdminExecutor-3460-thread-1-processing-n:127.0.0.1:42081_solr caeae54f-a8d3-45f5-916d-881a112a88202601118793163291 CREATE) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 981838 INFO  (qtp2126475454-18801) [n:127.0.0.1:42081_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : caeae54f-a8d3-45f5-916d-881a112a88202601118793163291
   [junit4]   2> 981838 INFO  (qtp2126475454-18801) [n:127.0.0.1:42081_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=caeae54f-a8d3-45f5-916d-881a112a88202601118793163291&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 981839 INFO  (qtp1566498685-18807) [n:127.0.0.1:32821_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : caeae54f-a8d3-45f5-916d-881a112a88202601118792265481
   [junit4]   2> 981839 INFO  (qtp1566498685-18807) [n:127.0.0.1:32821_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=caeae54f-a8d3-45f5-916d-881a112a88202601118792265481&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 981840 INFO  (OverseerThreadFactory-3459-thread-1-processing-n:127.0.0.1:32821_solr) [n:127.0.0.1:32821_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 983513 INFO  (OverseerCollectionConfigSetProcessor-99280484846338054-127.0.0.1:32821_solr-n_0000000000) [n:127.0.0.1:32821_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1069515 INFO  (TEST-AsyncCallRequestStatusResponseTest.testAsyncCallStatusResponse-seed#[DDA44F59B76C1CFD]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testAsyncCallStatusResponse
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=AsyncCallRequestStatusResponseTest -Dtests.method=testAsyncCallStatusResponse -Dtests.seed=DDA44F59B76C1CFD -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=de-AT -Dtests.timezone=ECT -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 90.0s J2 | AsyncCallRequestStatusResponseTest.testAsyncCallStatusResponse <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Expected collection 'asynccall' to have 2 shards and 1 replica
   [junit4]    > null
   [junit4]    > Live Nodes: [127.0.0.1:32821_solr, 127.0.0.1:42081_solr]
   [junit4]    > Last available state: DocCollection(asynccall//collections/asynccall/state.json/3)={
   [junit4]    >   "pullReplicas":"0",
   [junit4]    >   "replicationFactor":"1",
   [junit4]    >   "shards":{
   [junit4]    >     "shard1":{
   [junit4]    >       "range":"80000000-ffffffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{"core_node3":{
   [junit4]    >           "core":"asynccall_shard1_replica_n1",
   [junit4]    >           "base_url":"https://127.0.0.1:32821/solr",
   [junit4]    >           "node_name":"127.0.0.1:32821_solr",
   [junit4]    >           "state":"down",
   [junit4]    >           "type":"NRT",
   [junit4]    >           "leader":"true"}}},
   [junit4]    >     "shard2":{
   [junit4]    >       "range":"0-7fffffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{"core_node4":{
   [junit4]    >           "core":"asynccall_shard2_replica_n2",
   [junit4]    >           "base_url":"https://127.0.0.1:42081/solr",
   [junit4]    >           "node_name":"127.0.0.1:42081_solr",
   [junit4]    >           "state":"down",
   [junit4]    >           "type":"NRT",
   [junit4]    >           "leader":"true"}}}},
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1",
   [junit4]    >   "autoAddReplicas":"false",
   [junit4]    >   "nrtReplicas":"1",
   [junit4]    >   "tlogReplicas":"0"}
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DDA44F59B76C1CFD:87A52AB599569124]:0)
   [junit4]    > 	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:269)
   [junit4]    > 	at org.apache.solr.cloud.AsyncCallRequestStatusResponseTest.testAsyncCallStatusResponse(AsyncCallRequestStatusResponseTest.java:43)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1069520 INFO  (jetty-closer-5598-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@4d37a027{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1069520 INFO  (jetty-closer-5598-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1573226895
   [junit4]   2> 1069520 INFO  (jetty-closer-5598-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1069520 INFO  (jetty-closer-5598-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@71607478: rootName = solr_42081, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@19b626f5
   [junit4]   2> 1069520 INFO  (jetty-closer-5598-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@1c5068b0{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1069521 INFO  (jetty-closer-5598-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1486267429
   [junit4]   2> 1069521 INFO  (jetty-closer-5598-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1069521 INFO  (jetty-closer-5598-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7bc65335: rootName = solr_32821, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@c8965ad
   [junit4]   2> 1069539 INFO  (jetty-closer-5598-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1069539 INFO  (jetty-closer-5598-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5913c512: rootName = solr_42081, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@40c8f6c1
   [junit4]   2> 1069540 INFO  (jetty-closer-5598-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1069540 INFO  (jetty-closer-5598-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@35952279: rootName = solr_32821, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@40c8f6c1
   [junit4]   2> 1069545 INFO  (jetty-closer-5598-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1069545 INFO  (jetty-closer-5598-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1b8dc428: rootName = solr_42081, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@3ec6ff5d
   [junit4]   2> 1069546 INFO  (jetty-closer-5598-thread-1) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:42081_solr
   [junit4]   2> 1069546 INFO  (jetty-closer-5598-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1069547 INFO  (jetty-closer-5598-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1069547 INFO  (jetty-closer-5598-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@66a51adb: rootName = solr_32821, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@3ec6ff5d
   [junit4]   2> 1069547 INFO  (jetty-closer-5598-thread-2) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:32821_solr
   [junit4]   2> 1069547 INFO  (jetty-closer-5598-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1069553 INFO  (coreCloseExecutor-3474-thread-1) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.c.SolrCore [asynccall_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@46d6b4a2
   [junit4]   2> 1069554 INFO  (zkCallback-5615-thread-4-processing-n:127.0.0.1:32821_solr) [n:127.0.0.1:32821_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 1069554 INFO  (coreCloseExecutor-3474-thread-1) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.asynccall.shard1.replica_n1, tag=1188476066
   [junit4]   2> 1069554 INFO  (coreCloseExecutor-3474-thread-1) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3a3a7307: rootName = solr_32821, domain = solr.core.asynccall.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.asynccall.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@2f3fb7b5
   [junit4]   2> 1069554 INFO  (coreCloseExecutor-3475-thread-1) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.c.SolrCore [asynccall_shard2_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@1479b6bd
   [junit4]   2> 1069554 INFO  (coreCloseExecutor-3475-thread-1) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.asynccall.shard2.replica_n2, tag=343520957
   [junit4]   2> 1069554 INFO  (coreCloseExecutor-3475-thread-1) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@572f1138: rootName = solr_42081, domain = solr.core.asynccall.shard2.replica_n2, service url = null, agent id = null] for registry solr.core.asynccall.shard2.replica_n2 / com.codahale.metrics.MetricRegistry@3e24ddbd
   [junit4]   2> 1069555 INFO  (zkCallback-5613-thread-3-processing-n:127.0.0.1:42081_solr) [n:127.0.0.1:42081_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 1069562 INFO  (coreCloseExecutor-3474-thread-1) [n:127.0.0.1:32821_solr c:asynccall s:shard1 r:core_node3 x:asynccall_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.asynccall.shard1.leader, tag=1188476066
   [junit4]   2> 1069563 INFO  (coreCloseExecutor-3475-thread-1) [n:127.0.0.1:42081_solr c:asynccall s:shard2 r:core_node4 x:asynccall_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.asynccall.shard2.leader, tag=343520957
   [junit4]   2> 1069564 INFO  (jetty-closer-5598-thread-2) [    ] o.a.s.c.Overseer Overseer (id=99280484846338054-127.0.0.1:32821_solr-n_0000000000) closing
   [junit4]   2> 1069564 INFO  (OverseerStateUpdate-99280484846338054-127.0.0.1:32821_solr-n_0000000000) [n:127.0.0.1:32821_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:32821_solr
   [junit4]   2> 1069564 WARN  (OverseerAutoScalingTriggerThread-99280484846338054-127.0.0.1:32821_solr-n_0000000000) [n:127.0.0.1:32821_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1069565 INFO  (zkCallback-5613-thread-3-processing-n:127.0.0.1:42081_solr) [n:127.0.0.1:42081_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42081_solr
   [junit4]   2> 1069565 INFO  (jetty-closer-5598-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6d01640{/solr,null,UNAVAILABLE}
   [junit4]   2> 1071066 INFO  (jetty-closer-5598-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@669c6103{/solr,null,UNAVAILABLE}
   [junit4]   2> 1071067 ERROR (SUITE-AsyncCallRequestStatusResponseTest-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1071067 INFO  (SUITE-AsyncCallRequestStatusResponseTest-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40263 40263
   [junit4]   2> 1071132 INFO  (Thread-5642) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40263 40263
   [junit4]   2> 1071132 WARN  (Thread-5642) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/aliases.json
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 	2	/solr/configs/conf
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	9	/solr/collections/asynccall/state.json
   [junit4]   2> 	3	/solr/clusterstate.json
   [junit4]   2> 	3	/solr/clusterprops.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 	3	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.AsyncCallRequestStatusResponseTest_DDA44F59B76C1CFD-001
   [junit4]   2> Jan 02, 2018 1:34:28 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=535, maxMBSortInHeap=7.167919805329502, sim=RandomSimilarity(queryNorm=false): {}, locale=de-AT, timezone=ECT
   [junit4]   2> NOTE: Linux 4.10.0-40-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=8,threads=1,free=159524784,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [CollectionReloadTest, TestZkChroot, IndexSchemaTest, TestEmbeddedSolrServerConstructors, SolrCoreCheckLockOnStartupTest, TestSolrConfigHandlerConcurrent, DocValuesNotIndexedTest, DirectSolrConnectionTest, TestLeaderInitiatedRecoveryThread, TestPullReplica, TestJavabinTupleStreamParser, TestDistribDocBasedVersion, HdfsThreadLeakTest, InfixSuggestersTest, TestReloadDeadlock, TestHdfsUpdateLog, SolrJmxReporterTest, DocExpirationUpdateProcessorFactoryTest, MetricUtilsTest, JvmMetricsTest, HdfsBasicDistributedZkTest, TestSystemIdResolver, TestSchemalessBufferedUpdates, FileBasedSpellCheckerTest, TestMaxScoreQueryParser, BJQParserTest, OverseerTest, TestHdfsCloudBackupRestore, TestPolicyCloud, TestDistributedGrouping, TestNamedUpdateProcessors, TestJsonRequest, DistributedFacetPivotLargeTest, DistributedFacetSimpleRefinementLongTailTest, TestStressLiveNodes, SolrTestCaseJ4Test, TestNoOpRegenerator, PreAnalyzedUpdateProcessorTest, LargeFieldTest, TestQueryTypes, TestCollectionsAPIViaSolrCloudCluster, ZkStateReaderTest, FieldAnalysisRequestHandlerTest, TimeZoneUtilsTest, TestShardHandlerFactory, AnalyticsMergeStrategyTest, ConfigSetsAPITest, SmileWriterTest, RequiredFieldsTest, ScriptEngineTest, SuggesterTest, TestCloudPivotFacet, TestConfigSetsAPIZkFailure, BlockJoinFacetSimpleTest, TestMaxTokenLenTokenizer, ClusterStateTest, TestNumericRangeQuery64, DistanceFunctionTest, TestImpersonationWithHadoopAuth, TestInitParams, SpellCheckCollatorWithCollapseTest, TestRequestForwarding, HdfsWriteToMultipleCollectionsTest, CollectionsAPIAsyncDistributedZkTest, TestHalfAndHalfDocValues, SolrIndexMetricsTest, TestDistributedStatsComponentCardinality, TestClassicSimilarityFactory, TestLRUCache, PluginInfoTest, TestTriggerIntegration, TestRawTransformer, BlockCacheTest, TestPivotHelperCode, BadComponentTest, SSLMigrationTest, AutoAddReplicasIntegrationTest, LoggingHandlerTest, MoveReplicaHDFSTest, TestConfigOverlay, TestTolerantSearch, DeleteNodeTest, TestLegacyTerms, RequestLoggingTest, XsltUpdateRequestHandlerTest, AsyncCallRequestStatusResponseTest]
   [junit4] Completed [324/764 (1!)] on J2 in 92.81s, 1 test, 1 failure <<< FAILURES!

[...truncated 744 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_DDA44F59B76C1CFD-001/init-core-data-001
   [junit4]   2> 1958182 WARN  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=12 numCloses=12
   [junit4]   2> 1958182 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1958184 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1958184 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1958204 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_DDA44F59B76C1CFD-001/tempDir-001
   [junit4]   2> 1958204 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1958204 INFO  (Thread-8131) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1958204 INFO  (Thread-8131) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1958206 ERROR (Thread-8131) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1958304 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[DDA44F59B76C1CFD]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:39059
   [junit4]   2> 1958306 INFO  (zkConnectionManagerCallback-7670-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1958310 INFO  (jetty-launcher-7667-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1958310 INFO  (jetty-launcher-7667-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1958315 INFO  (jetty-launcher-7667-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@73a55c9e{/solr,null,AVAILABLE}
   [junit4]   2> 1958315 INFO  (jetty-launcher-7667-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@36454066{/solr,null,AVAILABLE}
   [junit4]   2> 1958319 INFO  (jetty-launcher-7667-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@58b5518{HTTP/1.1,[http/1.1]}{127.0.0.1:34687}
   [junit4]   2> 1958319 INFO  (jetty-launcher-7667-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7f90fbeb{HTTP/1.1,[http/1.1]}{127.0.0.1:39861}
   [junit4]   2> 1958319 INFO  (jetty-launcher-7667-thread-2) [    ] o.e.j.s.Server Started @1960218ms
   [junit4]   2> 1958319 INFO  (jetty-launcher-7667-thread-1) [    ] o.e.j.s.Server Started @1960218ms
   [junit4]   2> 1958319 INFO  (jetty-launcher-7667-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34687}
   [junit4]   2> 1958319 INFO  (jetty-launcher-7667-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39861}
   [junit4]   2> 1958320 ERROR (jetty-launcher-7667-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1958320 ERROR (jetty-launcher-7667-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1958320 INFO  (jetty-launcher-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 1958320 INFO  (jetty-launcher-7667-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 1958320 INFO  (jetty-launcher-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1958320 INFO  (jetty-launcher-7667-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1958320 INFO  (jetty-launcher-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1958320 INFO  (jetty-launcher-7667-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1958320 INFO  (jetty-launcher-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-02T13:49:15.524Z
   [junit4]   2> 1958320 INFO  (jetty-launcher-7667-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-02T13:49:15.524Z
   [junit4]   2> 1958322 INFO  (zkConnectionManagerCallback-7673-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1958322 INFO  (zkConnectionManagerCallback-7674-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1958323 INFO  (jetty-launcher-7667-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1958323 INFO  (jetty-launcher-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1958334 INFO  (jetty-launcher-7667-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39059/solr
   [junit4]   2> 1958334 INFO  (jetty-launcher-7667-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39059/solr
   [junit4]   2> 1958336 INFO  (zkConnectionManagerCallback-7680-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1958336 INFO  (zkConnectionManagerCallback-7682-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1958336 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x160b72221110004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1958337 INFO  (zkConnectionManagerCallback-7684-thread-1-processing-n:127.0.0.1:39861_solr) [n:127.0.0.1:39861_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1958338 INFO  (zkConnectionManagerCallback-7686-thread-1-processing-n:127.0.0.1:34687_solr) [n:127.0.0.1:34687_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1958384 INFO  (jetty-launcher-7667-thread-2) [n:127.0.0.1:34687_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1958385 INFO  (jetty-launcher-7667-thread-1) [n:127.0.0.1:39861_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1958385 INFO  (jetty-launcher-7667-thread-2) [n:127.0.0.1:34687_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34687_solr
   [junit4]   2> 1958385 INFO  (jetty-launcher-7667-thread-1) [n:127.0.0.1:39861_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39861_solr
   [junit4]   2> 1958385 INFO  (jetty-launcher-7667-thread-2) [n:127.0.0.1:34687_solr    ] o.a.s.c.Overseer Overseer (id=99280549013684230-127.0.0.1:34687_solr-n_0000000000) starting
   [junit4]   2> 1958388 INFO  (zkCallback-7683-thread-1-processing-n:127.0.0.1:39861_solr) [n:127.0.0.1:39861_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1958388 INFO  (zkCallback-7685-thread-1-processing-n:127.0.0.1:34687_solr) [n:127.0.0.1:34687_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1958390 INFO  (jetty-launcher-7667-thread-2) [n:127.0.0.1:34687_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34687_solr
   [junit4]   2> 1958390 INFO  (zkCallback-7683-thread-1-processing-n:127.0.0.1:39861_solr) [n:127.0.0.1:39861_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1958391 INFO  (zkCallback-7685-thread-1-processing-n:127.0.0.1:34687_solr) [n:127.0.0.1:34687_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1958425 INFO  (jetty-launcher-7667-thread-1) [n:127.0.0.1:39861_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
   [junit4]   2> 1958428 WARN  (jetty-launcher-7667-thread-1) [n:127.0.0.1:39861_solr    ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
   [junit4]   2> 1958430 INFO  (jetty-launcher-7667-thread-1) [n:127.0.0.1:39861_solr    ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=localhost.localdomain,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=jenkins, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
   [junit4]   2> 1958430 INFO  (jetty-launcher-7667-thread-2) [n:127.0.0.1:34687_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
   [junit4]   2> 1958430 WARN  (jetty-launcher-7667-thread-2) [n:127.0.0.1:34687_solr    ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
   [junit4]   2> 1958430 INFO  (jetty-launcher-7667-thread-2) [n:127.0.0.1:34687_solr    ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=localhost.localdomain,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=jenkins, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
   [junit4]   2> 1958433 INFO  (jetty-launcher-7667-thread-1) [n:127.0.0.1:39861_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 1958440 INFO  (jetty-launcher-7667-thread-2) [n:127.0.0.1:34687_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 1958441 INFO  (jetty-launcher-7667-thread-1-EventThread) [n:127.0.0.1:39861_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 1958446 INFO  (jetty-launcher-7667-thread-2-EventThread) [n:127.0.0.1:34687_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 1958453 WARN  (jetty-launcher-7667-thread-2) [n:127.0.0.1:34687_solr    ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
   [junit4]   2> 1958476 INFO  (jetty-launcher-7667-thread-2) [n:127.0.0.1:34687_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34687.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@100c4f86
   [junit4]   2> 1958480 INFO  (jetty-launcher-7667-thread-2) [n:127.0.0.1:34687_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34687.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@100c4f86
   [junit4]   2> 1958481 INFO  (jetty-launcher-7667-thread-2) [n:127.0.0.1:34687_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34687.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@100c4f86
   [junit4]   2> 1958481 INFO  (jetty-launcher-7667-thread-2) [n:127.0.0.1:34687_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_DDA44F59B76C1CFD-001/tempDir-001/node2/.
   [junit4]   2> 1959447 WARN  (jetty-launcher-7667-thread-1) [n:127.0.0.1:39861_solr    ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
   [junit4]   2> 1959478 INFO  (jetty-launcher-7667-thread-1) [n:127.0.0.1:39861_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39861.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@100c4f86
   [junit4]   2> 1959486 INFO  (jetty-launcher-7667-thread-1) [n:127.0.0.1:39861_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39861.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@100c4f86
   [junit4]   2> 1959486 INFO  (jetty-launcher-7667-thread-1) [n:127.0.0.1:39861_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39861.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@100c4f86
   [junit4]   2> 1959487 INFO  (jetty-launcher-7667-thread-1) [n:127.0.0.1:39861_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_DDA44F59B76C1CFD-001/tempDir-001/node1/.
   [junit4]   2> 1959518 INFO  (zkConnectionManagerCallback-7692-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1959530 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[DDA44F59B76C1CFD]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyWrongHost
   [junit4]   2> 1965342 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[DDA44F59B76C1CFD]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=wrongHost&p=wrongHost&t=test&e=1514936956737&s=gjMgA2evLKXsB/Ndal4FxYUQ/CM="; Path=/; Domain=127.0.0.1; Expires=Máirt, 02-Ean-2018 23:49:16 GMT; HttpOnly". Invalid 'expires' attribute: Máirt, 02-Ean-2018 23:49:16 GMT
   [junit4]   2> 1965343 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[DDA44F59B76C1CFD]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyWrongHost
   [junit4]   2> 1965385 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[DDA44F59B76C1CFD]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidGroup
   [junit4]   2> 1965423 WARN  (qtp897646469-28511) [    ] o.a.h.s.ShellBasedUnixGroupsMapping got exception trying to get groups for user bar: id: ‘bar’: no such user
   [junit4]   2> id: ‘bar’: no such user
   [junit4]   2> 
   [junit4]   2> 1965437 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[DDA44F59B76C1CFD]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=bogusGroup&p=bogusGroup&t=test&e=1514936962589&s=mkWdtnN1OTlREw5saBNvkY1DjAs="; Path=/; Domain=127.0.0.1; Expires=Máirt, 02-Ean-2018 23:49:22 GMT; HttpOnly". Invalid 'expires' attribute: Máirt, 02-Ean-2018 23:49:22 GMT
   [junit4]   2> 1965438 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[DDA44F59B76C1CFD]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidGroup
   [junit4]   2> 1965480 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[DDA44F59B76C1CFD]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidProxyUser
   [junit4]   2> 1965482 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[DDA44F59B76C1CFD]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=bar&p=bar&t=test&e=1514936962684&s=qosuAyV8JXzvLCkcZ1GzFdxkkZ8="; Path=/; Domain=127.0.0.1; Expires=Máirt, 02-Ean-2018 23:49:22 GMT; HttpOnly". Invalid 'expires' attribute: Máirt, 02-Ean-2018 23:49:22 GMT
   [junit4]   2> 1965483 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[DDA44F59B76C1CFD]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidProxyUser
   [junit4]   2> 1965528 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[DDA44F59B76C1CFD]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNullProxyUser
   [junit4]   2> 1965528 INFO  (qtp897646469-28498) [    ] o.a.s.s.SolrDispatchFilter Error authenticating
   [junit4]   2> java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
   [junit4]   2> 	at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:134)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilt

[...truncated too long message...]

keeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2> 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Jan 02, 2018 1:50:01 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1334, maxMBSortInHeap=6.608567518653556, sim=RandomSimilarity(queryNorm=false): {}, locale=ga-IE, timezone=Etc/GMT-8
   [junit4]   2> NOTE: Linux 4.10.0-40-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=8,threads=1,free=305783704,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [CollectionReloadTest, TestZkChroot, IndexSchemaTest, TestEmbeddedSolrServerConstructors, SolrCoreCheckLockOnStartupTest, TestSolrConfigHandlerConcurrent, DocValuesNotIndexedTest, DirectSolrConnectionTest, TestLeaderInitiatedRecoveryThread, TestPullReplica, TestJavabinTupleStreamParser, TestDistribDocBasedVersion, HdfsThreadLeakTest, InfixSuggestersTest, TestReloadDeadlock, TestHdfsUpdateLog, SolrJmxReporterTest, DocExpirationUpdateProcessorFactoryTest, MetricUtilsTest, JvmMetricsTest, HdfsBasicDistributedZkTest, TestSystemIdResolver, TestSchemalessBufferedUpdates, FileBasedSpellCheckerTest, TestMaxScoreQueryParser, BJQParserTest, OverseerTest, TestHdfsCloudBackupRestore, TestPolicyCloud, TestDistributedGrouping, TestNamedUpdateProcessors, TestJsonRequest, DistributedFacetPivotLargeTest, DistributedFacetSimpleRefinementLongTailTest, TestStressLiveNodes, SolrTestCaseJ4Test, TestNoOpRegenerator, PreAnalyzedUpdateProcessorTest, LargeFieldTest, TestQueryTypes, TestCollectionsAPIViaSolrCloudCluster, ZkStateReaderTest, FieldAnalysisRequestHandlerTest, TimeZoneUtilsTest, TestShardHandlerFactory, AnalyticsMergeStrategyTest, ConfigSetsAPITest, SmileWriterTest, RequiredFieldsTest, ScriptEngineTest, SuggesterTest, TestCloudPivotFacet, TestConfigSetsAPIZkFailure, BlockJoinFacetSimpleTest, TestMaxTokenLenTokenizer, ClusterStateTest, TestNumericRangeQuery64, DistanceFunctionTest, TestImpersonationWithHadoopAuth, TestInitParams, SpellCheckCollatorWithCollapseTest, TestRequestForwarding, HdfsWriteToMultipleCollectionsTest, CollectionsAPIAsyncDistributedZkTest, TestHalfAndHalfDocValues, SolrIndexMetricsTest, TestDistributedStatsComponentCardinality, TestClassicSimilarityFactory, TestLRUCache, PluginInfoTest, TestTriggerIntegration, TestRawTransformer, BlockCacheTest, TestPivotHelperCode, BadComponentTest, SSLMigrationTest, AutoAddReplicasIntegrationTest, LoggingHandlerTest, MoveReplicaHDFSTest, TestConfigOverlay, TestTolerantSearch, DeleteNodeTest, TestLegacyTerms, RequestLoggingTest, XsltUpdateRequestHandlerTest, AsyncCallRequestStatusResponseTest, TestFieldSortValues, TestFieldCacheSortRandom, TestGroupingSearch, TestCoreDiscovery, UnloadDistributedZkTest, TestDynamicLoading, TestFiltering, ReturnFieldsTest, BitVectorTest, TestTolerantUpdateProcessorRandomCloud, TestCloudDeleteByQuery, HighlighterTest, AlternateDirectoryTest, TestFieldResource, SuggesterWFSTTest, TestPayloadScoreQParserPlugin, TestComputePlanAction, TestCSVResponseWriter, LeaderElectionContextKeyTest, SolrMetricManagerTest, HdfsRecoveryZkTest, TestDistribIDF, TestUninvertingReader, TestValueSourceCache, TestGenericDistributedQueue, NodeLostTriggerTest, TestElisionMultitermQuery, TestMiniSolrCloudClusterSSL, TestTrieFacet, CopyFieldTest, TestRuleBasedAuthorizationPlugin, TestReplicationHandlerBackup, PKIAuthenticationIntegrationTest, EnumFieldTest, CSVRequestHandlerTest, TestComponentsName, TestLFUCache, TestHighlightDedupGrouping, TestEmbeddedSolrServerSchemaAPI, ConnectionReuseTest, AliasIntegrationTest, ChaosMonkeySafeLeaderWithPullReplicasTest, ConcurrentDeleteAndCreateCollectionTest, CustomCollectionTest, DeleteShardTest, DistribJoinFromCollectionTest, DistributedVersionInfoTest, LeaderInitiatedRecoveryOnCommitTest, MigrateRouteKeyTest, MoveReplicaTest, OverseerModifyCollectionTest, OverseerStatusTest, RemoteQueryErrorTest, ReplaceNodeTest, SharedFSAutoReplicaFailoverTest, SimpleCollectionCreateDeleteTest, SolrXmlInZkTest, TestAuthenticationFramework, TestCloudSearcherWarming, TestLocalFSCloudBackupRestore, TestPullReplicaErrorHandling, TestSSLRandomization, TestSegmentSorting, TestSkipOverseerOperations, TestSolrCloudWithSecureImpersonation]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSolrCloudWithSecureImpersonation -Dtests.seed=DDA44F59B76C1CFD -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ga-IE -Dtests.timezone=Etc/GMT-8 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | TestSolrCloudWithSecureImpersonation (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   [junit4]    >    1) Thread[id=28551, name=jetty-launcher-7667-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]    >    2) Thread[id=28547, name=jetty-launcher-7667-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DDA44F59B76C1CFD]:0)
   [junit4] Completed [561/764 (2!)] on J2 in 46.46s, 12 tests, 1 error <<< FAILURES!

[...truncated 48897 lines...]

[JENKINS-EA] Lucene-Solr-7.x-Linux (64bit/jdk-10-ea+37) - Build # 1111 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1111/
Java: 64bit/jdk-10-ea+37 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.schema.TestBulkSchemaConcurrent.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:34123/f_bff/z

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:34123/f_bff/z
	at __randomizedtesting.SeedInfo.seed([DD473CE93D60C635:55130333939CABCD]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:320)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.base/java.net.SocketInputStream.socketRead0(Native Method)
	at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 42 more




Build Log:
[...truncated 1717 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/core/test/temp/junit4-J2-20180103_175916_398935503688000613352.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/core/test/temp/junit4-J1-20180103_175916_398324742793156244998.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 9 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/core/test/temp/junit4-J0-20180103_175916_39814589568310395457242.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 287 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/test-framework/test/temp/junit4-J1-20180103_180629_30911751952037137369207.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 12 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/test-framework/test/temp/junit4-J0-20180103_180629_3094868051304605350776.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/test-framework/test/temp/junit4-J2-20180103_180629_30912042552798394831035.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 1042 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/common/test/temp/junit4-J1-20180103_180755_9157480416773937170319.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 18 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/common/test/temp/junit4-J2-20180103_180755_9157202143240735683359.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/common/test/temp/junit4-J0-20180103_180755_91417066914512174428821.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 223 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/icu/test/temp/junit4-J2-20180103_180942_81016440580964675382816.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/icu/test/temp/junit4-J0-20180103_180942_810926479439342370056.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 5 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/icu/test/temp/junit4-J1-20180103_180942_8103381957290218990692.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 243 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/kuromoji/test/temp/junit4-J0-20180103_180953_6936920889356594045138.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/kuromoji/test/temp/junit4-J2-20180103_180953_6931454114382629857023.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 9 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/kuromoji/test/temp/junit4-J1-20180103_180953_693643413499541740535.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 154 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/morfologik/test/temp/junit4-J1-20180103_181014_1099277612577954676043.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/morfologik/test/temp/junit4-J2-20180103_181014_1091630180058119960923.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/morfologik/test/temp/junit4-J0-20180103_181014_1093295734360895498915.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 164 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/opennlp/test/temp/junit4-J1-20180103_181017_50880835516052103932.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/opennlp/test/temp/junit4-J0-20180103_181017_50810289359511150904213.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/opennlp/test/temp/junit4-J2-20180103_181017_50818295228960362818525.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 171 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/phonetic/test/temp/junit4-J2-20180103_181020_19714808036026029066589.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/phonetic/test/temp/junit4-J1-20180103_181020_19717642251862056518664.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/phonetic/test/temp/junit4-J0-20180103_181020_197835107924746391918.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 153 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/smartcn/test/temp/junit4-J1-20180103_181031_13614131273127712973517.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/smartcn/test/temp/junit4-J0-20180103_181031_13616720972304915440284.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 154 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/stempel/test/temp/junit4-J2-20180103_181039_4856133295881242694621.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/stempel/test/temp/junit4-J1-20180103_181039_4855771129871728317321.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/stempel/test/temp/junit4-J0-20180103_181039_4853561337363411821540.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 169 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/analysis/uima/test/temp/junit4-J0-20180103_181042_4169227269152543084183.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 192 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/backward-codecs/test/temp/junit4-J2-20180103_181051_716639879780135674431.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 22 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/backward-codecs/test/temp/junit4-J0-20180103_181051_7163418352857694975551.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/backward-codecs/test/temp/junit4-J1-20180103_181051_71614736304638973801447.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 1306 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/benchmark/test/temp/junit4-J1-20180103_181249_26217587490937118666449.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/benchmark/test/temp/junit4-J2-20180103_181249_26218301002929556856333.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/benchmark/test/temp/junit4-J0-20180103_181249_26217744865565049533966.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 233 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/classification/test/temp/junit4-J0-20180103_181256_9491915508713549700166.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/classification/test/temp/junit4-J1-20180103_181256_94913316802001541547234.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/classification/test/temp/junit4-J2-20180103_181256_94917169259830827835199.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 251 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/codecs/test/temp/junit4-J2-20180103_181305_76515771552983881613721.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 8 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/codecs/test/temp/junit4-J0-20180103_181305_7659426548296536597065.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 7 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/codecs/test/temp/junit4-J1-20180103_181305_76511750981781166384036.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 211 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/demo/test/temp/junit4-J0-20180103_181504_5794040052968759843509.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/demo/test/temp/junit4-J2-20180103_181504_57918204779938874675336.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/demo/test/temp/junit4-J1-20180103_181504_5798070477914078711795.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 167 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/expressions/test/temp/junit4-J1-20180103_181507_52911302233806673267553.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/expressions/test/temp/junit4-J2-20180103_181507_5295642344504515261006.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/expressions/test/temp/junit4-J0-20180103_181507_5292794799803386336245.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 228 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/facet/test/temp/junit4-J0-20180103_181513_7797956036022733758048.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/facet/test/temp/junit4-J2-20180103_181513_7791699643056006878581.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/facet/test/temp/junit4-J1-20180103_181513_77918184558116100770086.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 179 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/grouping/test/temp/junit4-J2-20180103_181542_68418348690961957709142.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/grouping/test/temp/junit4-J1-20180103_181542_684908875924274500905.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/grouping/test/temp/junit4-J0-20180103_181542_68410864888522420642039.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 243 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/highlighter/test/temp/junit4-J2-20180103_181555_239713252032770238174.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/highlighter/test/temp/junit4-J1-20180103_181555_2397170878477848129853.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/highlighter/test/temp/junit4-J0-20180103_181555_2394197913158981845643.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 160 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/join/test/temp/junit4-J2-20180103_181618_11116893810381847382476.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/join/test/temp/junit4-J0-20180103_181618_1114913788418327688170.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/join/test/temp/junit4-J1-20180103_181618_11115628792851953629713.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 149 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/memory/test/temp/junit4-J0-20180103_181635_7674079287864826850452.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/memory/test/temp/junit4-J1-20180103_181635_76718101989592330147484.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 177 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/misc/test/temp/junit4-J0-20180103_181644_7075725568281415774578.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 8 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/misc/test/temp/junit4-J1-20180103_181644_7093237562600838828528.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/misc/test/temp/junit4-J2-20180103_181644_7095932645891740460156.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 332 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/queries/test/temp/junit4-J1-20180103_181700_52715437319135946282012.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/queries/test/temp/junit4-J0-20180103_181700_52717859056341263706495.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/queries/test/temp/junit4-J2-20180103_181700_5271263113710401671626.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 230 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/queryparser/test/temp/junit4-J2-20180103_181709_1125203808965828792674.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/queryparser/test/temp/junit4-J1-20180103_181709_1128591138878102980350.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/queryparser/test/temp/junit4-J0-20180103_181709_11212814475766472008495.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 207 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/replicator/test/temp/junit4-J1-20180103_181717_32415712439878457395694.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/replicator/test/temp/junit4-J2-20180103_181717_3246830691180550722329.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/replicator/test/temp/junit4-J0-20180103_181717_324979394306714166888.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 216 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/sandbox/test/temp/junit4-J0-20180103_181730_4325771861175597782868.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 7 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/sandbox/test/temp/junit4-J2-20180103_181730_432499156249389865168.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/sandbox/test/temp/junit4-J1-20180103_181730_43212497803253726412429.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 317 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/spatial-extras/test/temp/junit4-J0-20180103_181826_86517181256438851205158.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/spatial-extras/test/temp/junit4-J2-20180103_181826_86518062030649890246294.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/spatial-extras/test/temp/junit4-J1-20180103_181826_8655708868317284451272.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 148 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/spatial/test/temp/junit4-J0-20180103_181847_15818293327254701132241.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 166 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/spatial3d/test/temp/junit4-J2-20180103_181849_61213661495313664911051.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 9 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/spatial3d/test/temp/junit4-J0-20180103_181849_61114565982438499565022.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 8 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/spatial3d/test/temp/junit4-J1-20180103_181849_61110953012568624255860.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 244 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/suggest/test/temp/junit4-J0-20180103_182006_4967111863898800951297.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/suggest/test/temp/junit4-J2-20180103_182006_49612218586309658810649.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/lucene/build/suggest/test/temp/junit4-J1-20180103_182006_4969641028556289311311.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 4216 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/temp/junit4-J1-20180103_182105_1539113933369053468094.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 5 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/temp/junit4-J2-20180103_182105_1558275460996352219756.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 6 lines...]
   [junit4] Suite: org.apache.solr.schema.TestBulkSchemaConcurrent
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestBulkSchemaConcurrent_DD473CE93D60C635-001/init-core-data-001
   [junit4]   2> 2516562 WARN  (SUITE-TestBulkSchemaConcurrent-seed#[DD473CE93D60C635]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=98 numCloses=98
   [junit4]   2> 2516562 INFO  (SUITE-TestBulkSchemaConcurrent-seed#[DD473CE93D60C635]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2516563 INFO  (SUITE-TestBulkSchemaConcurrent-seed#[DD473CE93D60C635]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
   [junit4]   2> 2516563 INFO  (SUITE-TestBulkSchemaConcurrent-seed#[DD473CE93D60C635]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2516563 INFO  (SUITE-TestBulkSchemaConcurrent-seed#[DD473CE93D60C635]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /f_bff/z
   [junit4]   2> 2516564 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2516565 INFO  (Thread-5923) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2516565 INFO  (Thread-5923) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2516566 ERROR (Thread-5923) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2516665 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.ZkTestServer start zk server on port:43485
   [junit4]   2> 2516666 INFO  (zkConnectionManagerCallback-4278-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2516667 INFO  (zkConnectionManagerCallback-4280-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2516669 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-managed-schema.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2516670 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2516670 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2516671 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2516671 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2516672 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2516672 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2516673 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2516674 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2516674 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2516675 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2516675 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 2516755 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2516760 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3ff92929{/f_bff/z,null,AVAILABLE}
   [junit4]   2> 2516760 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@15c5621b{HTTP/1.1,[http/1.1]}{127.0.0.1:34123}
   [junit4]   2> 2516760 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.e.j.s.Server Started @2518236ms
   [junit4]   2> 2516760 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/f_bff/z, solr.data.dir=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestBulkSchemaConcurrent_DD473CE93D60C635-001/tempDir-001/control/data, hostPort=34123, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestBulkSchemaConcurrent_DD473CE93D60C635-001/control-001/cores, replicaType=NRT}
   [junit4]   2> 2516760 ERROR (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2516761 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 2516761 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2516761 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2516761 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-03T19:03:03.429337Z
   [junit4]   2> 2516764 INFO  (zkConnectionManagerCallback-4282-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2516764 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2516764 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestBulkSchemaConcurrent_DD473CE93D60C635-001/control-001/solr.xml
   [junit4]   2> 2516765 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2516766 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2516766 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@63701bd0, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2516768 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43485/solr
   [junit4]   2> 2516770 INFO  (zkConnectionManagerCallback-4286-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2516776 INFO  (zkConnectionManagerCallback-4288-thread-1-processing-n:127.0.0.1:34123_f_bff%2Fz) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2516817 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2516817 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34123_f_bff%2Fz
   [junit4]   2> 2516817 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.Overseer Overseer (id=99287445224423428-127.0.0.1:34123_f_bff%2Fz-n_0000000000) starting
   [junit4]   2> 2516821 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34123_f_bff%2Fz
   [junit4]   2> 2516824 INFO  (zkCallback-4287-thread-1-processing-n:127.0.0.1:34123_f_bff%2Fz) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2516851 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@63701bd0
   [junit4]   2> 2516859 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@63701bd0
   [junit4]   2> 2516859 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@63701bd0
   [junit4]   2> 2516861 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestBulkSchemaConcurrent_DD473CE93D60C635-001/control-001/cores
   [junit4]   2> 2516943 INFO  (zkConnectionManagerCallback-4294-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2516944 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2516944 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43485/solr ready
   [junit4]   2> 2516946 INFO  (qtp504628049-22118) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:34123_f_bff%252Fz&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2516952 INFO  (OverseerThreadFactory-8929-thread-1-processing-n:127.0.0.1:34123_f_bff%2Fz) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2517055 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2517055 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2517163 INFO  (zkCallback-4287-thread-1-processing-n:127.0.0.1:34123_f_bff%2Fz) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2518066 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 2518070 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
   [junit4]   2> 2518075 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2518138 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2518146 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.ManagedIndexSchema Created and persisted managed schema znode at /configs/conf1/managed-schema
   [junit4]   2> 2518147 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/conf1/schema.xml to /configs/conf1/schema.xml.bak
   [junit4]   2> 2518147 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2518147 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@63701bd0
   [junit4]   2> 2518148 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2518148 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestBulkSchemaConcurrent_DD473CE93D60C635-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestBulkSchemaConcurrent_DD473CE93D60C635-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2518150 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=10, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 2518168 WARN  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, registerPath=/solr,/v2, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2518193 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2518193 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2518194 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2518194 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2518195 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=37, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=65.201171875, floorSegmentMB=0.6591796875, forceMergeDeletesPctAllowed=6.992688834809096, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.32511861208922477
   [junit4]   2> 2518195 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2a95fba9[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2518196 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2518196 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2518196 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 2518196 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 2518196 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2518197 INFO  (searcherExecutor-8932-thread-1-processing-n:127.0.0.1:34123_f_bff%2Fz x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@2a95fba9[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2518197 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588599125301002240
   [junit4]   2> 2518200 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2518200 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2518200 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34123/f_bff/z/control_collection_shard1_replica_n1/
   [junit4]   2> 2518200 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2518200 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:34123/f_bff/z/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2518200 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2518201 INFO  (qtp504628049-22114) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:34123/f_bff/z/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2576975 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.ChaosMonkey monkey: stop jetty! 34123
   [junit4]   2> 2576975 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=797427443
   [junit4]   2> 2576975 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2576975 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@756b4caf: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@42e6401c
   [junit4]   2> 2576980 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2576980 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6be45325: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@1f47d3ee
   [junit4]   2> 2576984 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2576984 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@58545c2a: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@517e6d0e
   [junit4]   2> 2576985 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:34123_f_bff%2Fz
   [junit4]   2> 2576985 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 2576985 INFO  (zkCallback-4287-thread-2-processing-n:127.0.0.1:34123_f_bff%2Fz) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 2576985 INFO  (coreCloseExecutor-8937-thread-1) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@180beb0f
   [junit4]   2> 2576985 INFO  (coreCloseExecutor-8937-thread-1) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.control_collection.shard1.replica_n1, tag=403434255
   [junit4]   2> 2576985 INFO  (coreCloseExecutor-8937-thread-1) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3470fe47: rootName = null, domain = solr.core.control_collection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.control_collection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@ce8ad6a
   [junit4]   2> 2576993 INFO  (coreCloseExecutor-8937-thread-1) [n:127.0.0.1:34123_f_bff%2Fz c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.control_collection.shard1.leader, tag=403434255
   [junit4]   2> 2587858 INFO  (ScheduledTrigger-8927-thread-3) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 2587860 INFO  (AutoscalingActionExecutor-8928-thread-1-processing-n:127.0.0.1:34123_f_bff%2Fz) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 2707396500 sessionWrapper.createTime 2707396500830485, this.sessionWrapper.createTime 2707396500830485 
   [junit4]   2> 2587861 INFO  (AutoscalingActionExecutor-8928-thread-1-processing-n:127.0.0.1:34123_f_bff%2Fz) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.a.ExecutePlanAction No operations to execute for event: {
   [junit4]   2>   "id":"99e5a89ef88f6Ten8i9mhq9yg9ybgjuyrr13z41",
   [junit4]   2>   "source":".auto_add_replicas",
   [junit4]   2>   "eventTime":2707386488817910,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2707386488817910],
   [junit4]   2>     "_enqueue_time_":2707396497803378,
   [junit4]   2>     "nodeNames":["127.0.0.1:34123_f_bff%2Fz"]}}
   [junit4]   2> 2696947 ERROR (qtp504628049-22118) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: create the collection time out:180s
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:314)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:246)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:224)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> 
   [junit4]   2> 2696948 INFO  (qtp504628049-22118) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:34123_f_bff%252Fz&wt=javabin&version=2} status=500 QTime=180001
   [junit4]   2> 2696948 ERROR (qtp504628049-22118) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: create the collection time out:180s
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:314)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:246)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:224)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> 
   [junit4]   2> 3117111 ERROR (OverseerThreadFactory-8929-thread-1-processing-n:127.0.0.1:34123_f_bff%2Fz) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: http://127.0.0.1:34123/f_bff/z
   [junit4]   2> org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:34123/f_bff/z
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]   2> 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]   2> 	at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:172)
   [junit4]   2> 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   [junit4]   2> 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
   [junit4]   2> 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   [junit4]   2> 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
   [junit4]   2> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]   2> 	at java.base/java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]   2> 	at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]   2> 	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]   2> 	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]   2> 	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   [junit4]   2> 	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   [junit4]   2> 	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]   2> 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]   2> 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]   2> 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]   2> 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:118)
   [junit4]   2> 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]   2> 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]   2> 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]   2> 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
   [junit4]   2> 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]   2> 	... 12 more
   [junit4]   2> 3117112 INFO  (TEST-TestBulkSchemaConcurrent.test-seed#[DD473CE93D60C635]) [    ] o.a.s.c.Overseer Overseer (id=99287445224423428-127.0.0.1:34123_f_bff%2Fz-n_0000000000) closing
   [junit4]   2> 3117113 ERROR (OverseerThreadFactory-8929-thread-1-processing-n:127.0.0.1:34123_f_bff%2Fz) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [control_collection].
   [junit4]   2> 3117113 INFO  (OverseerStateUpdate-99287445224423428-127.0.0.1:34123_f_bff%2Fz-n_0000000000) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:34123_f_bff%2Fz
   [junit4]   2> 3117113 INFO  (OverseerThreadFactory-8929-thread-1-processing-n:127.0.0.1:34123_f_bff%2Fz) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 3147148 INFO  (zkCallback-4287-thread-5-processing-n:127.0.0.1:34123_f_bff%2Fz) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 3147149 ERROR (OverseerThreadFactory-8929-thread-1-processing-n:127.0.0.1:34123_f_bff%2Fz) [n:127.0.0.1:34123_f_bff%2Fz    ] o.a.s.c.OverseerCollectionMessageHandler Collection: control_collection operation: create failed:org.apache.solr.common.SolrException: Could not fully remove collection: control_collection
   [junit4]   2> 	at org.apache.solr.cloud.DeleteCollectionCmd.call(DeleteCollectionCmd.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionMessageHandler.cleanupCollection(OverseerCollectionMessageHandler.java:688)
   [junit4]   2> 	at org.apache.solr.cloud.CreateCollectionCmd.call(CreateCollectionCmd.java:240)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:242)
   [junit4]   2> 	at org.

[...truncated too long message...]

unit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-analytics/test/temp/junit4-J2-20180103_191707_58010520674255162101255.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-analytics/test/temp/junit4-J0-20180103_191707_5805033098079046067153.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 553 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-clustering/test/temp/junit4-J2-20180103_191809_38810202903621054370230.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-clustering/test/temp/junit4-J1-20180103_191809_38815049840890266815413.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-clustering/test/temp/junit4-J0-20180103_191809_3885387748453200629545.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 1260 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-dataimporthandler-extras/test/temp/junit4-J1-20180103_191826_5989464180509198861864.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-dataimporthandler-extras/test/temp/junit4-J0-20180103_191826_5983803071759418542198.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 644 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J2-20180103_191834_85814571812300070469707.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20180103_191834_8589186496075421198087.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J1-20180103_191834_85813026770035460989216.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 558 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-cell/test/temp/junit4-J0-20180103_191856_81515139690172974847151.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-cell/test/temp/junit4-J2-20180103_191856_81511498045993882875824.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-cell/test/temp/junit4-J1-20180103_191856_81510750936892542936952.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 560 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-langid/test/temp/junit4-J1-20180103_191908_85417775499574394221260.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-langid/test/temp/junit4-J0-20180103_191908_85113464410790225928344.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 659 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/test/temp/junit4-J0-20180103_191916_8073311441736449204845.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/test/temp/junit4-J1-20180103_191916_80713674978312454700862.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-ltr/test/temp/junit4-J2-20180103_191916_80715005718313165152193.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J2: EOF ----

[...truncated 596 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-uima/test/temp/junit4-J1-20180103_191948_1596467374670332942772.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-uima/test/temp/junit4-J0-20180103_191948_1588278713076320134694.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 548 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/contrib/solr-velocity/test/temp/junit4-J0-20180103_192001_63910237688329483165731.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 32274 lines...]

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-9.0.1) - Build # 1110 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1110/
Java: 64bit/jdk-9.0.1 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.SyncSliceTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:41401/_ipa/m

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:41401/_ipa/m
	at __randomizedtesting.SeedInfo.seed([9B8EA4AD00B43A2F:13DA9B77AE4857D7]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:320)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.base/java.net.SocketInputStream.socketRead0(Native Method)
	at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 42 more




Build Log:
[...truncated 13406 lines...]
   [junit4] Suite: org.apache.solr.cloud.SyncSliceTest
   [junit4]   2> 1180173 INFO  (SUITE-SyncSliceTest-seed#[9B8EA4AD00B43A2F]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.SyncSliceTest_9B8EA4AD00B43A2F-001/init-core-data-001
   [junit4]   2> 1180173 WARN  (SUITE-SyncSliceTest-seed#[9B8EA4AD00B43A2F]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 1180173 INFO  (SUITE-SyncSliceTest-seed#[9B8EA4AD00B43A2F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1180174 INFO  (SUITE-SyncSliceTest-seed#[9B8EA4AD00B43A2F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
   [junit4]   2> 1180174 INFO  (SUITE-SyncSliceTest-seed#[9B8EA4AD00B43A2F]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /_ipa/m
   [junit4]   2> 1180175 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1180175 INFO  (Thread-3316) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1180175 INFO  (Thread-3316) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1180179 ERROR (Thread-3316) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1180275 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.ZkTestServer start zk server on port:35447
   [junit4]   2> 1180276 INFO  (zkConnectionManagerCallback-7116-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1180278 INFO  (zkConnectionManagerCallback-7118-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1180279 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1180280 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1180280 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1180281 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1180281 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1180281 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1180282 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1180282 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1180282 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1180283 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1180283 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1180284 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 1180360 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1180361 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7f86daf{/_ipa/m,null,AVAILABLE}
   [junit4]   2> 1180362 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@cf28a10{HTTP/1.1,[http/1.1]}{127.0.0.1:41401}
   [junit4]   2> 1180362 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.e.j.s.Server Started @1182344ms
   [junit4]   2> 1180362 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/_ipa/m, solr.data.dir=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.SyncSliceTest_9B8EA4AD00B43A2F-001/tempDir-001/control/data, hostPort=41401, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.SyncSliceTest_9B8EA4AD00B43A2F-001/control-001/cores}
   [junit4]   2> 1180362 ERROR (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1180362 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 1180362 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1180362 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1180362 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-03T15:06:03.496748Z
   [junit4]   2> 1180363 INFO  (zkConnectionManagerCallback-7120-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1180364 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1180364 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.SyncSliceTest_9B8EA4AD00B43A2F-001/control-001/solr.xml
   [junit4]   2> 1180365 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1180365 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1180366 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@972bf87, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1180367 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35447/solr
   [junit4]   2> 1180368 INFO  (zkConnectionManagerCallback-7124-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1180370 INFO  (zkConnectionManagerCallback-7126-thread-1-processing-n:127.0.0.1:41401__ipa%2Fm) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1180392 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1180392 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41401__ipa%2Fm
   [junit4]   2> 1180392 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.c.Overseer Overseer (id=99286513307484164-127.0.0.1:41401__ipa%2Fm-n_0000000000) starting
   [junit4]   2> 1180394 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41401__ipa%2Fm
   [junit4]   2> 1180395 INFO  (zkCallback-7125-thread-1-processing-n:127.0.0.1:41401__ipa%2Fm) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1180427 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@972bf87
   [junit4]   2> 1180432 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@972bf87
   [junit4]   2> 1180433 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@972bf87
   [junit4]   2> 1180433 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.SyncSliceTest_9B8EA4AD00B43A2F-001/control-001/cores
   [junit4]   2> 1180443 INFO  (zkConnectionManagerCallback-7132-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1180443 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1180444 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35447/solr ready
   [junit4]   2> 1180445 INFO  (qtp918805636-20520) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:41401__ipa%252Fm&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1180447 INFO  (OverseerThreadFactory-4834-thread-1-processing-n:127.0.0.1:41401__ipa%2Fm) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 1180550 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1180550 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1180652 INFO  (zkCallback-7125-thread-1-processing-n:127.0.0.1:41401__ipa%2Fm) [n:127.0.0.1:41401__ipa%2Fm    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1181565 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 1181575 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 1181660 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1181668 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 1181669 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@972bf87
   [junit4]   2> 1181669 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1181669 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.SyncSliceTest_9B8EA4AD00B43A2F-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.SyncSliceTest_9B8EA4AD00B43A2F-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 1181670 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=606598773, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1181672 WARN  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1181693 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1181693 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1181694 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1181694 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1181695 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=10.2685546875, floorSegmentMB=0.740234375, forceMergeDeletesPctAllowed=12.012774393322573, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1181695 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6638c3bd[control_collection_shard1_replica_n1] main]
   [junit4]   2> 1181696 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1181696 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1181696 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1181697 INFO  (searcherExecutor-4837-thread-1-processing-n:127.0.0.1:41401__ipa%2Fm x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@6638c3bd[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1181697 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588584214514630656
   [junit4]   2> 1181699 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1181699 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1181699 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:41401/_ipa/m/control_collection_shard1_replica_n1/
   [junit4]   2> 1181699 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1181699 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:41401/_ipa/m/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 1181699 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1181700 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:41401/_ipa/m/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 1240503 INFO  (zkConnectionManagerCallback-7134-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   1> / (2)
   [junit4]   1> DATA:
   [junit4]   1>     
   [junit4]   1>  /zookeeper (1)
   [junit4]   1>  DATA:
   [junit4]   1>      
   [junit4]   1>  /solr (11)
   [junit4]   1>   /solr/configs (2)
   [junit4]   1>    /solr/configs/_default (7)
   [junit4]   1>     /solr/configs/_default/protwords.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/_default/managed-schema (0)
   [junit4]   1>     DATA:
   [junit4]   1>         <?xml version="1.0" encoding="UTF-8" ?>
   [junit4]   1>         <!--
   [junit4]   1>          Licensed to the Apache Software Foundation (ASF) under one or more
   [junit4]   1>          contributor license agreements.  See the NOTICE file distributed with
   [junit4]   1>          this work for additional information regarding copyright ownership.
   [junit4]   1>          The ASF licenses this file to You under the Apache License, Version 2.0
   [junit4]   1>          (the "License"); you may not use this file except in compliance with
   [junit4]   1>          the License.  You may obtain a copy of the License at
   [junit4]   1>         
   [junit4]   1>              http://www.apache.org/licenses/LICENSE-2.0
   [junit4]   1>         
   [junit4]   1>          Unless required by applicable law or agreed to in writing, software
   [junit4]   1>          distributed under the License is distributed on an "AS IS" BASIS,
   [junit4]   1>          WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   [junit4]   1>          See the License for the specific language governing permissions and
   [junit4]   1>          limitations under the License.
   [junit4]   1>         -->
   [junit4]   1>         
   [junit4]   1>         <!--
   [junit4]   1>         
   [junit4]   1>          This example schema is the recommended starting point for users.
   [junit4]   1>          It should be kept correct and concise, usable out-of-the-box.
   [junit4]   1>         
   [junit4]   1>         
   [junit4]   1>          For more information, on how to customize this file, please see
   [junit4]   1>          http://lucene.apache.org/solr/guide/documents-fields-and-schema-design.html
   [junit4]   1>         
   [junit4]   1>          PERFORMANCE NOTE: this schema includes many optional features and should not
   [junit4]   1>          be used for benchmarking.  To improve performance one could
   [junit4]   1>           - set stored="false" for all fields possible (esp large fields) when you
   [junit4]   1>             only need to search on the field but don't need to return the original
   [junit4]   1>             value.
   [junit4]   1>           - set indexed="false" if you don't need to search on the field, but only
   [junit4]   1>             return the field as a result of searching on other indexed fields.
   [junit4]   1>           - remove all unneeded copyField statements
   [junit4]   1>           - for best index size and searching performance, set "index" to false
   [junit4]   1>             for all general text fields, use copyField to copy them to the
   [junit4]   1>             catchall "text" field, and use that for searching.
   [junit4]   1>         -->
   [junit4]   1>         
   [junit4]   1>         <schema name="default-config" version="1.6">
   [junit4]   1>             <!-- attribute "name" is the name of this schema and is only used for display purposes.
   [junit4]   1>                version="x.y" is Solr's version number for the schema syntax and 
   [junit4]   1>                semantics.  It should not normally be changed by applications.
   [junit4]   1>         
   [junit4]   1>                1.0: multiValued attribute did not exist, all fields are multiValued 
   [junit4]   1>                     by nature
   [junit4]   1>                1.1: multiValued attribute introduced, false by default 
   [junit4]   1>                1.2: omitTermFreqAndPositions attribute introduced, true by default 
   [junit4]   1>                     except for text fields.
   [junit4]   1>                1.3: removed optional field compress feature
   [junit4]   1>                1.4: autoGeneratePhraseQueries attribute introduced to drive QueryParser
   [junit4]   1>                     behavior when a single string produces multiple tokens.  Defaults 
   [junit4]   1>                     to off for version >= 1.4
   [junit4]   1>                1.5: omitNorms defaults to true for primitive field types 
   [junit4]   1>                     (int, float, boolean, string...)
   [junit4]   1>                1.6: useDocValuesAsStored defaults to true.
   [junit4]   1>             -->
   [junit4]   1>         
   [junit4]   1>             <!-- Valid attributes for fields:
   [junit4]   1>              name: mandatory - the name for the field
   [junit4]   1>              type: mandatory - the name of a field type from the 
   [junit4]   1>                fieldTypes section
   [junit4]   1>              indexed: true if this field should be indexed (searchable or sortable)
   [junit4]   1>              stored: true if this field should be retrievable
   [junit4]   1>              docValues: true if this field should have doc values. Doc Values is
   [junit4]   1>                recommended (required, if you are using *Point fields) for faceting,
   [junit4]   1>                grouping, sorting and function queries. Doc Values will make the index
   [junit4]   1>                faster to load, more NRT-friendly and more memory-efficient. 
   [junit4]   1>                They are currently only supported by StrField, UUIDField, all 
   [junit4]   1>                *PointFields, and depending on the field type, they might require
   [junit4]   1>                the field to be single-valued, be required or have a default value
   [junit4]   1>                (check the documentation of the field type you're interested in for
   [junit4]   1>                more information)
   [junit4]   1>              multiValued: true if this field may contain multiple values per document
   [junit4]   1>              omitNorms: (expert) set to true to omit the norms associated with
   [junit4]   1>                this field (this disables length normalization and index-time
   [junit4]   1>                boosting for the field, and saves some memory).  Only full-text
   [junit4]   1>                fields or fields that need an index-time boost need norms.
   [junit4]   1>                Norms are omitted for primitive (non-analyzed) types by default.
   [junit4]   1>              termVectors: [false] set to true to store the term vector for a
   [junit4]   1>                given field.
   [junit4]   1>                When using MoreLikeThis, fields used for similarity should be
   [junit4]   1>                stored for best performance.
   [junit4]   1>              termPositions: Store position information with the term vector.  
   [junit4]   1>                This will increase storage costs.
   [junit4]   1>              termOffsets: Store offset information with the term vector. This 
   [junit4]   1>                will increase storage costs.
   [junit4]   1>              required: The field is required.  It will throw an error if the
   [junit4]   1>                value does not exist
   [junit4]   1>              default: a value that should be used if no value is specified
   [junit4]   1>                when adding a document.
   [junit4]   1>             -->
   [junit4]   1>         
   [junit4]   1>             <!-- field names should consist of alphanumeric or underscore characters only and
   [junit4]   1>               not start with a digit.  This is not currently strictly enforced,
   [junit4]   1>               but other field names will not have first class support from all components
   [junit4]   1>               and back compatibility is not guaranteed.  Names with both leading and
   [junit4]   1>               trailing underscores (e.g. _version_) are reserved.
   [junit4]   1>             -->
   [junit4]   1>         
   [junit4]   1>             <!-- In this _default configset, only four fields are pre-declared:
   [junit4]   1>                  id, _version_, and _text_ and _root_. All other fields will be type guessed and added via the
   [junit4]   1>                  "add-unknown-fields-to-the-schema" update request processor chain declared in solrconfig.xml.
   [junit4]   1>                  
   [junit4]   1>                  Note that many dynamic fields are also defined - you can use them to specify a 
   [junit4]   1>                  field's type via field naming conventions - see below.
   [junit4]   1>           
   [junit4]   1>                  WARNING: The _text_ catch-all field will significantly increase your index size.
   [junit4]   1>                  If you don't need it, consider removing it and the corresponding copyField directive.
   [junit4]   1>             -->
   [junit4]   1>         
   [junit4]   1>             <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
   [junit4]   1>             <!-- docValues are enabled by default for long type so we don't need to index the version field  -->
   [junit4]   1>             <field name="_version_" type="plong" indexed="false" stored="false"/>
   [junit4]   1>             <field name="_root_" type="string" indexed="true" stored="false" docValues="false" />
   [junit4]   1>             <field name="_text_" type="text_general" indexed="true" stored="false" multiValued="true"/>
   [junit4]   1>         
   [junit4]   1>             <!-- This can be enabled, in case the client does not know what fields may be searched. It isn't enabled by default
   [junit4]   1>                  because it's very expensive to index everything twice. -->
   [junit4]   1>             <!-- <copyField source="*" dest="_text_"/> -->
   [junit4]   1>         
   [junit4]   1>             <!-- Dynamic field definitions allow using convention over configuration
   [junit4]   1>                for fields via the specification of patterns to match field names.
   [junit4]   1>                EXAMPLE:  name="*_i" will match any field ending in _i (like myid_i, z_i)
   [junit4]   1>                RESTRICTION: the glob-like pattern in the name attribute must have a "*" only at the start or the end.  -->
   [junit4]   1>            
   [junit4]   1>             <dynamicField name="*_i"  type="pint"    indexed="true"  stored="true"/>
   [junit4]   1>             <dynamicField name="*_is" type="pints"    indexed="true"  stored="true"/>
   [junit4]   1>             <dynamicField name="*_s"  type="string"  indexed="true"  stored="true" />
   [junit4]   1>             <dynamicField name="*_ss" type="strings"  indexed="true"  stored="true"/>
   [junit4]   1>             <dynamicField name="*_l"  type="plong"   indexed="true"  stored="true"/>
   [junit4]   1>             <dynamicField name="*_ls" type="plongs"   indexed="true"  stored="true"/>
   [junit4]   1>             <dynamicField name="*_txt" type="text_general" indexed="true" stored="true"/>
   [junit4]   1>             <dynamicField name="*_b"  type="boolean" indexed="true" stored="true"/>
   [junit4]   1>             <dynamicField name="*_bs" type="booleans" indexed="true" stored="true"/>
   [junit4]   1>             <dynamicField name="*_f"  type="pfloat"  indexed="true"  stored="true"/>
   [junit4]   1>             <dynamicField name="*_fs" type="pfloats"  indexed="true"  stored="true"/>
   [junit4]   1>             <dynamicField name="*_d"  type="pdouble" indexed="true"  stored="true"/>
   [junit4]   1>             <dynamicField name="*_ds" type="pdoubles" indexed="true"  stored="true"/>
   [junit4]   1>         
   [junit4]   1>             <!-- Type used for data-driven schema, to add a string copy for each text field -->
   [junit4]   1>             <dynamicField name="*_str" type="strings" stored="false" docValues="true" indexed="false" />
   [junit4]   1>         
   [junit4]   1>             <dynamicField name="*_dt"  type="pdate"    indexed="true"  stored="true"/>
   [junit4]   1>             <dynamicField name="*_dts" type="pdate"    indexed="true"  stored="true" multiValued="true"/>
   [junit4]   1>             <dynamicField name="*_p"  type="location" indexed="true" stored="true"/>
   [junit4]   1>             <dynamicField name="*_srpt"  type="location_rpt" indexed="true" stored="true"/>
   [junit4]   1>             
   [junit4]   1>             <!-- payloaded dynamic fields -->
   [junit4]   1>             <dynamicField name="*_dpf" type="delimited_payloads_float" indexed="true"  stored="true"/>
   [junit4]   1>             <dynamicField name="*_dpi" type="delimited_payloads_int" indexed="true"  stored="true"/>
   [junit4]   1>             <dynamicField name="*_dps" type="delimited_payloads_string" indexed="true"  stored="true"/>
   [junit4]   1>         
   [junit4]   1>             <dynamicField name="attr_*" type="text_general" indexed="true" stored="true" multiValued="true"/>
   [junit4]   1>         
   [junit4]   1>             <!-- Field to use to determine and enforce document uniqueness.
   [junit4]   1>               Unless this field is marked with required="false", it will be a required field
   [junit4]   1>             -->
   [junit4]   1>             <uniqueKey>id</uniqueKey>
   [junit4]   1>         
   [junit4]   1>             <!-- copyField commands copy one field to another at the time a document
   [junit4]   1>                is added to the index.  It's used either to index the same field differently,
   [junit4]   1>                or to add multiple fields to the same field for easier/faster searching.
   [junit4]   1>         
   [junit4]   1>             <copyField source="sourceFieldName" dest="destinationFieldName"/>
   [junit4]   1>             -->
   [junit4]   1>         
   [junit4]   1>             <!-- field type definitions. The "name" attribute is
   [junit4]   1>                just a label to be used by field definitions.  The "class"
   [junit4]   1>                attribute and any other attributes determine the real
   [junit4]   1>                behavior of the fieldType.
   [junit4]   1>                  Class names starting with "solr" refer to java classes in a
   [junit4]   1>                standard package such as org.apache.solr.analysis
   [junit4]   1>             -->
   [junit4]   1>         
   [junit4]   1>             <!-- sortMissingLast and sortMissingFirst attributes are optional attributes are
   [junit4]   1>                  currently supported on types that are sorted internally as strings
   [junit4]   1>                  and on numeric types.
   [junit4]   1>                This includes "string", "boolean", "pint", "pfloat", "plong", "pdate", "pdouble".
   [junit4]   1>                - If sortMissingLast="true", then a sort on this field will cause documents
   [junit4]   1>                  without the field to come after documents with the field,
   [junit4]   1>                  regardless of the requested sort order (asc or desc).
   [junit4]   1>                - If sortMissingFirst="true", then a sort on this field will cause documents
   [junit4]   1>                  without the field to come before documents with the field,
   [junit4]   1>                  regardless of the requested sort order.
   [junit4]   1>                - If sortMissingLast="false" and sortMissingFirst="false" (the default),
   [junit4]   1>                  then default lucene sorting will be used which places docs without the
   [junit4]   1>                  field first in an ascending sort and last in a descending sort.
   [junit4]   1>             -->
   [junit4]   1>         
   [junit4]   1>             <!-- The StrField type is not analyzed, but indexed/stored verbatim. -->
   [junit4]   1>             <fieldType name="string" class="solr.StrField" sortMissingLast="true" docValues="true" />
   [junit4]   1>             <fieldType name="strings" class="solr.StrField" sortMissingLast="true" multiValued="true" docValues="true" />
   [junit4]   1>         
   [junit4]   1>             <!-- boolean type: "true" or "false" -->
   [junit4]   1>             <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true"/>
   [junit4]   1>             <fieldType name="booleans" class="solr.BoolField" sortMissingLast="true" multiValued="true"/>
   [junit4]   1>         
   [junit4]   1>             <!--
   [junit4]   1>               Numeric field types that index values using KD-trees.
   [junit4]   1>               Point fields don't support FieldCache, so they must have docValues="true" if needed for sorting, faceting, functions, etc.
   [junit4]   1>             -->
   [junit4]   1>             <fieldType name="pint" class="solr.IntPointField" docValues="true"/>
   [junit4]   1>             <fieldType name="pfloat" class="solr.FloatPointField" docValues="true"/>
   [junit4]   1>             <fieldType name="plong" class="solr.LongPointField" docValues="true"/>
   [junit4]   1>             <fieldType name="pdouble" class="solr.DoublePointField" docValues="true"/>
   [junit4]   1>             
   [junit4]   1>             <fieldType name="pints" class="solr.IntPointField" docValues="true" multiValued="true"/>
   [junit4]   1>             <fieldType name="pfloats" class="solr.FloatPointField" docValues="true" multiValued="true"/>
   [junit4]   1>             <fieldType name="plongs" class="solr.LongPointField" docValues="true" multiValued="true"/>
   [junit4]   1>             <fieldType name="pdoubles" class="solr.DoublePointField" docValues="true" multiValued="true"/>
   [junit4]   1>         
   [junit4]   1>             <!-- The format for this date field is of the form 1995-12-31T23:59:59Z, and
   [junit4]   1>                  is a more restricted form of the canonical representation of dateTime
   [junit4]   1>                  http://www.w3.org/TR/xmlschema-2/#dateTime    
   [junit4]   1>                  The trailing "Z" designates UTC time and is mandatory.
   [junit4]   1>                  Optional fractional seconds are allowed: 1995-12-31T23:59:59.999Z
   [junit4]   1>                  All other components are mandatory.
   [junit4]   1>         
   [junit4]   1>                  Expressions can also be used to denote calculations that should be
   [junit4]   1>                  performed relative to "NOW" to determine the value, ie...
   [junit4]   1>         
   [junit4]   1>                        NOW/HOUR
   [junit4]   1>                           ... Round to the start of the current hour
   [junit4]   1>                        NOW-1DAY
   [junit4]   1>                           ... Exactly 1 day prior to now
   [junit4]   1>                        NOW/DAY+6MONTHS+3DAYS
   [junit4]   1>                           ... 6 months and 3 days in the future from the start of
   [junit4]   1>                               the current day
   [junit4]   1>                               
   [junit4]   1>               -->
   [junit4]   1>             <!-- KD-tree versions of date fields -->
   [junit4]   1>             <fieldType name="pdate" class="solr.DatePointField" docValues="true"/>
   [junit4]   1>             <fieldType name="pdates" class="solr.DatePointField" docValues="true" multiValued="true"/>
   [junit4]   1>             
   [junit4]   1>             <!--Binary data type. The data should be sent/retrieved in as Base64 encoded Strings -->
   [junit4]   1>             <fieldType name="binary" class="solr.BinaryField"/>
   [junit4]   1>         
   [junit4]   1>             <!-- solr.TextField allows the specification of custom text analyzers
   [junit4]   1>                  specified as a tokenizer and a list of token filters. Different
   [junit4]   1>                  analyzers may be specified for indexing and querying.
   [junit4]   1>         
   [junit4]   1>                  The optional positionIncrementGap puts space between multiple fields of
   [junit4]   1>                  this type on the same document, with the purpose of preventing false phrase
   [junit4]   1>                  matching across fields.
   [junit4]   1>         
   [junit4]   1>                  For more info on customizing your analyzer chain, please see
   [junit4]   1>                  http://lucene.apache.org/solr/guide/understanding-analyzers-tokenizers-and-filters.html#understanding-analyzers-tokenizers-and-filters
   [junit4]   1>              -->
   [junit4]   1>         
   [junit4]   1>             <!-- One can also specify an existing Analyzer class that has a
   [junit4]   1>                  default constructor via the class attribute on the analyzer element.
   [junit4]   1>                  Example:
   [junit4]   1>             <fieldType name="text_greek" class="solr.TextField">
   [junit4]   1>               <analyzer class="org.apache.lucene.analysis.el.GreekAnalyzer"/>
   [junit4]   1>             </fieldType>
   [junit4]   1>             -->
   [junit4]   1>         
   [junit4]   1>             <!-- A text field that only splits on whitespace for exact matching of words -->
   [junit4]   1>             <dynamicField name="*_ws" type="text_ws"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_ws" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer>
   [junit4]   1>                 <tokenizer class="solr.WhitespaceTokenizerFactory"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>         
   [junit4]   1>             <!-- A general text field that has reasonable, generic
   [junit4]   1>                  cross-language defaults: it tokenizes with StandardTokenizer,
   [junit4]   1>         	       removes stop words from case-insensitive "stopwords.txt"
   [junit4]   1>         	       (empty by default), and down cases.  At query time only, it
   [junit4]   1>         	       also applies synonyms.
   [junit4]   1>         	  -->
   [junit4]   1>             <fieldType name="text_general" class="solr.TextField" positionIncrementGap="100" multiValued="true">
   [junit4]   1>               <analyzer type="index">
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
   [junit4]   1>                 <!-- in this example, we will only use synonyms at query time
   [junit4]   1>                 <filter class="solr.SynonymGraphFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
   [junit4]   1>                 <filter class="solr.FlattenGraphFilterFactory"/>
   [junit4]   1>                 -->
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>               <analyzer type="query">
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
   [junit4]   1>                 <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>         
   [junit4]   1>             <!-- A text field with defaults appropriate for English: it tokenizes with StandardTokenizer,
   [junit4]   1>                  removes English stop words (lang/stopwords_en.txt), down cases, protects words from protwords.txt, and
   [junit4]   1>                  finally applies Porter's stemming.  The query time analyzer also applies synonyms from synonyms.txt. -->
   [junit4]   1>             <dynamicField name="*_txt_en" type="text_en"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_en" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer type="index">
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <!-- in this example, we will only use synonyms at query time
   [junit4]   1>                 <filter class="solr.SynonymGraphFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
   [junit4]   1>                 <filter class="solr.FlattenGraphFilterFactory"/>
   [junit4]   1>                 -->
   [junit4]   1>                 <!-- Case insensitive stop word removal.
   [junit4]   1>                 -->
   [junit4]   1>                 <filter class="solr.StopFilterFactory"
   [junit4]   1>                         ignoreCase="true"
   [junit4]   1>                         words="lang/stopwords_en.txt"
   [junit4]   1>                     />
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.EnglishPossessiveFilterFactory"/>
   [junit4]   1>                 <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
   [junit4]   1>                 <!-- Optionally you may want to use this less aggressive stemmer instead of PorterStemFilterFactory:
   [junit4]   1>                 <filter class="solr.EnglishMinimalStemFilterFactory"/>
   [junit4]   1>         	      -->
   [junit4]   1>                 <filter class="solr.PorterStemFilterFactory"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>               <analyzer type="query">
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory"
   [junit4]   1>                         ignoreCase="true"
   [junit4]   1>                         words="lang/stopwords_en.txt"
   [junit4]   1>                 />
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.EnglishPossessiveFilterFactory"/>
   [junit4]   1>                 <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
   [junit4]   1>                 <!-- Optionally you may want to use this less aggressive stemmer instead of PorterStemFilterFactory:
   [junit4]   1>                 <filter class="solr.EnglishMinimalStemFilterFactory"/>
   [junit4]   1>         	      -->
   [junit4]   1>                 <filter class="solr.PorterStemFilterFactory"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>         
   [junit4]   1>             <!-- A text field with defaults appropriate for English, plus
   [junit4]   1>                  aggressive word-splitting and autophrase features enabled.
   [junit4]   1>                  This field is just like text_en, except it adds
   [junit4]   1>                  WordDelimiterGraphFilter to enable splitting and matching of
   [junit4]   1>                  words on case-change, alpha numeric boundaries, and
   [junit4]   1>                  non-alphanumeric chars.  This means certain compound word
   [junit4]   1>                  cases will work, for example query "wi fi" will match
   [junit4]   1>                  document "WiFi" or "wi-fi".
   [junit4]   1>             -->
   [junit4]   1>             <dynamicField name="*_txt_en_split" type="text_en_splitting"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_en_splitting" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true">
   [junit4]   1>               <analyzer type="index">
   [junit4]   1>                 <tokenizer class="solr.WhitespaceTokenizerFactory"/>
   [junit4]   1>                 <!-- in this example, we will only use synonyms at query time
   [junit4]   1>                 <filter class="solr.SynonymGraphFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
   [junit4]   1>                 -->
   [junit4]   1>                 <!-- Case insensitive stop word removal.
   [junit4]   1>                 -->
   [junit4]   1>                 <filter class="solr.StopFilterFactory"
   [junit4]   1>                         ignoreCase="true"
   [junit4]   1>                         words="lang/stopwords_en.txt"
   [junit4]   1>                 />
   [junit4]   1>                 <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
   [junit4]   1>                 <filter class="solr.PorterStemFilterFactory"/>
   [junit4]   1>                 <filter class="solr.FlattenGraphFilterFactory" />
   [junit4]   1>               </analyzer>
   [junit4]   1>               <analyzer type="query">
   [junit4]   1>                 <tokenizer class="solr.WhitespaceTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory"
   [junit4]   1>                         ignoreCase="true"
   [junit4]   1>                         words="lang/stopwords_en.txt"
   [junit4]   1>                 />
   [junit4]   1>                 <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
   [junit4]   1>                 <filter class="solr.PorterStemFilterFactory"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>         
   [junit4]   1>             <!-- Less flexible matching, but less false matches.  Probably not ideal for product names,
   [junit4]   1>                  but may be good for SKUs.  Can insert dashes in the wrong place and still match. -->
   [junit4]   1>             <dynamicField name="*_txt_en_split_tight" type="text_en_splitting_tight"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_en_splitting_tight" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true">
   [junit4]   1>               <analyzer type="index">
   [junit4]   1>                 <tokenizer class="solr.WhitespaceTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="false"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>
   [junit4]   1>                 <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="0" generateNumberParts="0" catenateWords="1" catenateNumbers="1" catenateAll="0"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
   [junit4]   1>                 <filter class="solr.EnglishMinimalStemFilterFactory"/>
   [junit4]   1>                 <!-- this filter can remove any duplicate tokens that appear at the same position - sometimes
   [junit4]   1>                      possible with WordDelimiterGraphFilter in conjuncton with stemming. -->
   [junit4]   1>                 <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
   [junit4]   1>                 <filter class="solr.FlattenGraphFilterFactory" />
   [junit4]   1>               </analyzer>
   [junit4]   1>               <analyzer type="query">
   [junit4]   1>                 <tokenizer class="solr.WhitespaceTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="false"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>
   [junit4]   1>                 <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="0" generateNumberParts="0" catenateWords="1" catenateNumbers="1" catenateAll="0"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
   [junit4]   1>                 <filter class="solr.EnglishMinimalStemFilterFactory"/>
   [junit4]   1>                 <!-- this filter can remove any duplicate tokens that appear at the same position - sometimes
   [junit4]   1>                      possible with WordDelimiterGraphFilter in conjuncton with stemming. -->
   [junit4]   1>                 <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>         
   [junit4]   1>             <!-- Just like text_general except it reverses the characters of
   [junit4]   1>         	       each token, to enable more efficient leading wildcard queries.
   [junit4]   1>             -->
   [junit4]   1>             <dynamicField name="*_txt_rev" type="text_general_rev"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_general_rev" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer type="index">
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.ReversedWildcardFilterFactory" withOriginal="true"
   [junit4]   1>                         maxPosAsterisk="3" maxPosQuestion="2" maxFractionAsterisk="0.33"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>               <analyzer type="query">
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>         
   [junit4]   1>             <dynamicField name="*_phon_en" type="phonetic_en"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="phonetic_en" stored="false" indexed="true" class="solr.TextField" >
   [junit4]   1>               <analyzer>
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.DoubleMetaphoneFilterFactory" inject="false"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>         
   [junit4]   1>             <!-- lowercases the entire field value, keeping it as a single token.  -->
   [junit4]   1>             <dynamicField name="*_s_lower" type="lowercase"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="lowercase" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer>
   [junit4]   1>                 <tokenizer class="solr.KeywordTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory" />
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>         
   [junit4]   1>             <!-- 
   [junit4]   1>               Example of using PathHierarchyTokenizerFactory at index time, so
   [junit4]   1>               queries for paths match documents at that path, or in descendent paths
   [junit4]   1>             -->
   [junit4]   1>             <dynamicField name="*_descendent_path" type="descendent_path"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="descendent_path" class="solr.TextField">
   [junit4]   1>               <analyzer type="index">
   [junit4]   1>                 <tokenizer class="solr.PathHierarchyTokenizerFactory" delimiter="/" />
   [junit4]   1>               </analyzer>
   [junit4]   1>               <analyzer type="query">
   [junit4]   1>                 <tokenizer class="solr.KeywordTokenizerFactory" />
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>         
   [junit4]   1>             <!--
   [junit4]   1>               Example of using PathHierarchyTokenizerFactory at query time, so
   [junit4]   1>               queries for paths match documents at that path, or in ancestor paths
   [junit4]   1>             -->
   [junit4]   1>             <dynamicField name="*_ancestor_path" type="ancestor_path"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="ancestor_path" class="solr.TextField">
   [junit4]   1>               <analyzer type="index">
   [junit4]   1>                 <tokenizer class="solr.KeywordTokenizerFactory" />
   [junit4]   1>               </analyzer>
   [junit4]   1>               <analyzer type="query">
   [junit4]   1>                 <tokenizer class="solr.PathHierarchyTokenizerFactory" delimiter="/" />
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>         
   [junit4]   1>             <!-- This point type indexes the coordinates as separate fields (subFields)
   [junit4]   1>               If subFieldType is defined, it references a type, and a dynamic field
   [junit4]   1>               definition is created matching *___<typename>.  Alternately, if 
   [junit4]   1>               subFieldSuffix is defined, that is used to create the subFields.
   [junit4]   1>               Example: if subFieldType="double", then the coordinates would be
   [junit4]   1>                 indexed in fields myloc_0___double,myloc_1___double.
   [junit4]   1>               Example: if subFieldSuffix="_d" then the coordinates would be indexed
   [junit4]   1>                 in fields myloc_0_d,myloc_1_d
   [junit4]   1>               The subFields are an implementation detail of the fieldType, and end
   [junit4]   1>               users normally should not need to know about them.
   [junit4]   1>              -->
   [junit4]   1>             <dynamicField name="*_point" type="point"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="point" class="solr.PointType" dimension="2" subFieldSuffix="_d"/>
   [junit4]   1>         
   [junit4]   1>             <!-- A specialized field for geospatial search filters and distance sorting. -->
   [junit4]   1>             <fieldType name="location" class="solr.LatLonPointSpatialField" docValues="true"/>
   [junit4]   1>         
   [junit4]   1>             <!-- A geospatial field type that supports multiValued and polygon shapes.
   [junit4]   1>               For more information about this and other spatial fields see:
   [junit4]   1>               http://lucene.apache.org/solr/guide/spatial-search.html
   [junit4]   1>             -->
   [junit4]   1>             <fieldType name="location_rpt" class="solr.SpatialRecursivePrefixTreeFieldType"
   [junit4]   1>                        geo="true" distErrPct="0.025" maxDistErr="0.001" distanceUnits="kilometers" />
   [junit4]   1>         
   [junit4]   1>             <!-- Payloaded field types -->
   [junit4]   1>             <fieldType name="delimited_payloads_float" stored="false" indexed="true" class="solr.TextField">
   [junit4]   1>               <analyzer>
   [junit4]   1>                 <tokenizer class="solr.WhitespaceTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.DelimitedPayloadTokenFilterFactory" encoder="float"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             <fieldType name="delimited_payloads_int" stored="false" indexed="true" class="solr.TextField">
   [junit4]   1>               <analyzer>
   [junit4]   1>                 <tokenizer class="solr.WhitespaceTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.DelimitedPayloadTokenFilterFactory" encoder="integer"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             <fieldType name="delimited_payloads_string" stored="false" indexed="true" class="solr.TextField">
   [junit4]   1>               <analyzer>
   [junit4]   1>                 <tokenizer class="solr.WhitespaceTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.DelimitedPayloadTokenFilterFactory" encoder="identity"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>         
   [junit4]   1>             <!-- some examples for different languages (generally ordered by ISO code) -->
   [junit4]   1>         
   [junit4]   1>             <!-- Arabic -->
   [junit4]   1>             <dynamicField name="*_txt_ar" type="text_ar"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_ar" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <!-- for any non-arabic -->
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_ar.txt" />
   [junit4]   1>                 <!-- normalizes ﻯ to ﻱ, etc -->
   [junit4]   1>                 <filter class="solr.ArabicNormalizationFilterFactory"/>
   [junit4]   1>                 <filter class="solr.ArabicStemFilterFactory"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>         
   [junit4]   1>             <!-- Bulgarian -->
   [junit4]   1>             <dynamicField name="*_txt_bg" type="text_bg"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_bg" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/> 
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_bg.txt" /> 
   [junit4]   1>                 <filter class="solr.BulgarianStemFilterFactory"/>       
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Catalan -->
   [junit4]   1>             <dynamicField name="*_txt_ca" type="text_ca"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_ca" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <!-- removes l', etc -->
   [junit4]   1>                 <filter class="solr.ElisionFilterFactory" ignoreCase="true" articles="lang/contractions_ca.txt"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_ca.txt" />
   [junit4]   1>                 <filter class="solr.SnowballPorterFilterFactory" language="Catalan"/>       
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- CJK bigram (see text_ja for a Japanese configuration using morphological analysis) -->
   [junit4]   1>             <dynamicField name="*_txt_cjk" type="text_cjk"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_cjk" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer>
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <!-- normalize width before bigram, as e.g. half-width dakuten combine  -->
   [junit4]   1>                 <filter class="solr.CJKWidthFilterFactory"/>
   [junit4]   1>                 <!-- for any non-CJK -->
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.CJKBigramFilterFactory"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>         
   [junit4]   1>             <!-- Czech -->
   [junit4]   1>             <dynamicField name="*_txt_cz" type="text_cz"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_cz" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_cz.txt" />
   [junit4]   1>                 <filter class="solr.CzechStemFilterFactory"/>       
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Danish -->
   [junit4]   1>             <dynamicField name="*_txt_da" type="text_da"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_da" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_da.txt" format="snowball" />
   [junit4]   1>                 <filter class="solr.SnowballPorterFilterFactory" language="Danish"/>       
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- German -->
   [junit4]   1>             <dynamicField name="*_txt_de" type="text_de"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_de" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_de.txt" format="snowball" />
   [junit4]   1>                 <filter class="solr.GermanNormalizationFilterFactory"/>
   [junit4]   1>                 <filter class="solr.GermanLightStemFilterFactory"/>
   [junit4]   1>                 <!-- less aggressive: <filter class="solr.GermanMinimalStemFilterFactory"/> -->
   [junit4]   1>                 <!-- more aggressive: <filter class="solr.SnowballPorterFilterFactory" language="German2"/> -->
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Greek -->
   [junit4]   1>             <dynamicField name="*_txt_el" type="text_el"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_el" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <!-- greek specific lowercase for sigma -->
   [junit4]   1>                 <filter class="solr.GreekLowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="false" words="lang/stopwords_el.txt" />
   [junit4]   1>                 <filter class="solr.GreekStemFilterFactory"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Spanish -->
   [junit4]   1>             <dynamicField name="*_txt_es" type="text_es"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_es" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_es.txt" format="snowball" />
   [junit4]   1>                 <filter class="solr.SpanishLightStemFilterFactory"/>
   [junit4]   1>                 <!-- more aggressive: <filter class="solr.SnowballPorterFilterFactory" language="Spanish"/> -->
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Basque -->
   [junit4]   1>             <dynamicField name="*_txt_eu" type="text_eu"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_eu" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_eu.txt" />
   [junit4]   1>                 <filter class="solr.SnowballPorterFilterFactory" language="Basque"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Persian -->
   [junit4]   1>             <dynamicField name="*_txt_fa" type="text_fa"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_fa" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer>
   [junit4]   1>                 <!-- for ZWNJ -->
   [junit4]   1>                 <charFilter class="solr.PersianCharFilterFactory"/>
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.ArabicNormalizationFilterFactory"/>
   [junit4]   1>                 <filter class="solr.PersianNormalizationFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_fa.txt" />
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Finnish -->
   [junit4]   1>             <dynamicField name="*_txt_fi" type="text_fi"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_fi" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_fi.txt" format="snowball" />
   [junit4]   1>                 <filter class="solr.SnowballPorterFilterFactory" language="Finnish"/>
   [junit4]   1>                 <!-- less aggressive: <filter class="solr.FinnishLightStemFilterFactory"/> -->
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- French -->
   [junit4]   1>             <dynamicField name="*_txt_fr" type="text_fr"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_fr" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <!-- removes l', etc -->
   [junit4]   1>                 <filter class="solr.ElisionFilterFactory" ignoreCase="true" articles="lang/contractions_fr.txt"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_fr.txt" format="snowball" />
   [junit4]   1>                 <filter class="solr.FrenchLightStemFilterFactory"/>
   [junit4]   1>                 <!-- less aggressive: <filter class="solr.FrenchMinimalStemFilterFactory"/> -->
   [junit4]   1>                 <!-- more aggressive: <filter class="solr.SnowballPorterFilterFactory" language="French"/> -->
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Irish -->
   [junit4]   1>             <dynamicField name="*_txt_ga" type="text_ga"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_ga" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <!-- removes d', etc -->
   [junit4]   1>                 <filter class="solr.ElisionFilterFactory" ignoreCase="true" articles="lang/contractions_ga.txt"/>
   [junit4]   1>                 <!-- removes n-, etc. position increments is intentionally false! -->
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/hyphenations_ga.txt"/>
   [junit4]   1>                 <filter class="solr.IrishLowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_ga.txt"/>
   [junit4]   1>                 <filter class="solr.SnowballPorterFilterFactory" language="Irish"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Galician -->
   [junit4]   1>             <dynamicField name="*_txt_gl" type="text_gl"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_gl" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_gl.txt" />
   [junit4]   1>                 <filter class="solr.GalicianStemFilterFactory"/>
   [junit4]   1>                 <!-- less aggressive: <filter class="solr.GalicianMinimalStemFilterFactory"/> -->
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Hindi -->
   [junit4]   1>             <dynamicField name="*_txt_hi" type="text_hi"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_hi" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <!-- normalizes unicode representation -->
   [junit4]   1>                 <filter class="solr.IndicNormalizationFilterFactory"/>
   [junit4]   1>                 <!-- normalizes variation in spelling -->
   [junit4]   1>                 <filter class="solr.HindiNormalizationFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_hi.txt" />
   [junit4]   1>                 <filter class="solr.HindiStemFilterFactory"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Hungarian -->
   [junit4]   1>             <dynamicField name="*_txt_hu" type="text_hu"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_hu" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_hu.txt" format="snowball" />
   [junit4]   1>                 <filter class="solr.SnowballPorterFilterFactory" language="Hungarian"/>
   [junit4]   1>                 <!-- less aggressive: <filter class="solr.HungarianLightStemFilterFactory"/> -->   
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Armenian -->
   [junit4]   1>             <dynamicField name="*_txt_hy" type="text_hy"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_hy" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_hy.txt" />
   [junit4]   1>                 <filter class="solr.SnowballPorterFilterFactory" language="Armenian"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Indonesian -->
   [junit4]   1>             <dynamicField name="*_txt_id" type="text_id"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_id" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_id.txt" />
   [junit4]   1>                 <!-- for a less aggressive approach (only inflectional suffixes), set stemDerivational to false -->
   [junit4]   1>                 <filter class="solr.IndonesianStemFilterFactory" stemDerivational="true"/>
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Italian -->
   [junit4]   1>           <dynamicField name="*_txt_it" type="text_it"  indexed="true"  stored="true"/>
   [junit4]   1>           <fieldType name="text_it" class="solr.TextField" positionIncrementGap="100">
   [junit4]   1>               <analyzer> 
   [junit4]   1>                 <tokenizer class="solr.StandardTokenizerFactory"/>
   [junit4]   1>                 <!-- removes l', etc -->
   [junit4]   1>                 <filter class="solr.ElisionFilterFactory" ignoreCase="true" articles="lang/contractions_it.txt"/>
   [junit4]   1>                 <filter class="solr.LowerCaseFilterFactory"/>
   [junit4]   1>                 <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_it.txt" format="snowball" />
   [junit4]   1>                 <filter class="solr.ItalianLightStemFilterFactory"/>
   [junit4]   1>                 <!-- more aggressive: <filter class="solr.SnowballPorterFilterFactory" language="Italian"/> -->
   [junit4]   1>               </analyzer>
   [junit4]   1>             </fieldType>
   [junit4]   1>             
   [junit4]   1>             <!-- Japanese using morphological analysis (see text_cjk for a configuration using bigramming)
   [junit4]   1>         
   [junit4]   1>                  NOTE: If you want to optimize search for precision, use default operator AND in your request
   [junit4]   1>                  handler config (q.op) Use OR if you would like to optimize for recall (default).
   [junit4]   1>             -->
   [junit4]   1>             <dynamicField name="*_txt_ja" type="text_ja"  indexed="true"  stored="true"/>
   [junit4]   1>             <fieldType name="text_ja" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="false">
   [junit4]   1>               <analyzer>
   [junit4]   1>                 <!-- Kuromoji Japanese morphological analyzer/tokenizer (JapaneseTokenizer)
   [junit4]   1>         
   [junit4]   1>                    Kuromoji has a search mode (default) that does segmentation useful for search.  A heuristic
   [junit4]   1>                    is used to segment compounds into its parts and the compound itself is k

[...truncated too long message...]


   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1178)
   [junit4]   2> 	... 41 more
   [junit4]   2> 
   [junit4]   2> 1810678 INFO  (qtp918805636-20522) [n:127.0.0.1:41401__ipa%2Fm c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=630127
   [junit4]   2> 1810678 ERROR (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1810678 INFO  (TEST-SyncSliceTest.test-seed#[9B8EA4AD00B43A2F]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35447 35447
   [junit4]   2> 1815821 INFO  (Thread-3316) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35447 35447
   [junit4]   2> 1815822 WARN  (Thread-3316) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/aliases.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/clusterprops.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/live_nodes
   [junit4]   2> 	2	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.method=test -Dtests.seed=9B8EA4AD00B43A2F -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mk -Dtests.timezone=America/Campo_Grande -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR    636s J2 | SyncSliceTest.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:41401/_ipa/m
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9B8EA4AD00B43A2F:13DA9B77AE4857D7]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:320)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
   [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > 	at java.base/java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > 	at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]    > 	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]    > 	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]    > 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]    > 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]    > 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]    > 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]    > 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]    > 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
   [junit4]    > 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]    > 	... 42 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.SyncSliceTest_9B8EA4AD00B43A2F-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1010, maxMBSortInHeap=5.303558266449008, sim=RandomSimilarity(queryNorm=false): {}, locale=mk, timezone=America/Campo_Grande
   [junit4]   2> NOTE: Linux 4.10.0-40-generic amd64/Oracle Corporation 9.0.1 (64-bit)/cpus=8,threads=1,free=208100216,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [DistributedDebugComponentTest, TestSolrConfigHandlerConcurrent, AutoAddReplicasIntegrationTest, JvmMetricsTest, AliasIntegrationTest, SolrTestCaseJ4Test, CollectionReloadTest, MigrateRouteKeyTest, TestJavabinTupleStreamParser, TestConfigOverlay, TestSolrFieldCacheBean, TestLRUStatsCache, DeleteStatusTest, HdfsCollectionsAPIDistributedZkTest, DistributedFacetSimpleRefinementLongTailTest, DefaultValueUpdateProcessorTest, TestMaxTokenLenTokenizer, V2StandaloneTest, LeaderInitiatedRecoveryOnCommitTest, TestDistribIDF, SolrCoreCheckLockOnStartupTest, UninvertDocValuesMergePolicyTest, TestMaxScoreQueryParser, SolrIndexMetricsTest, TestPolicyCloud, SolrShardReporterTest, SharedFSAutoReplicaFailoverTest, TestCharFilters, LargeFieldTest, HdfsWriteToMultipleCollectionsTest, TestNamedUpdateProcessors, BJQParserTest, OverseerStatusTest, TestFuzzyAnalyzedSuggestions, CdcrUpdateLogTest, NodeMutatorTest, TestPartialUpdateDeduplication, TestHdfsUpdateLog, TestNestedDocsSort, TestNodeLostTrigger, TestTriggerIntegration, DisMaxRequestHandlerTest, CoreAdminRequestStatusTest, TestCloudSearcherWarming, TestValueSourceCache, TestReloadDeadlock, AtomicUpdateProcessorFactoryTest, TestTrieFacet, TestReload, TestRestManager, CollectionsAPIAsyncDistributedZkTest, TestSegmentSorting, TestUninvertingReader, SolrMetricManagerTest, InfixSuggestersTest, DistanceFunctionTest, TestLMJelinekMercerSimilarityFactory, TestConfigSetsAPIZkFailure, TestMiniSolrCloudClusterSSL, TestSystemIdResolver, DistribJoinFromCollectionTest, MetricUtilsTest, TestStressLiveNodes, TestSchemaNameResource, AsyncCallRequestStatusResponseTest, TestLFUCache, SortSpecParsingTest, TestLegacyTerms, XmlUpdateRequestHandlerTest, TestHdfsCloudBackupRestore, ClassificationUpdateProcessorIntegrationTest, CustomCollectionTest, BitVectorTest, UniqFieldsUpdateProcessorFactoryTest, ConnectionReuseTest, LeaderElectionContextKeyTest, TestDistributedStatsComponentCardinality, HdfsBasicDistributedZkTest, TestFiltering, ChaosMonkeyNothingIsSafeTest, FieldAnalysisRequestHandlerTest, TestInitParams, SystemInfoHandlerTest, DistributedFacetPivotLargeTest, TestSchemaVersionResource, TestManagedStopFilterFactory, TestComplexPhraseLeadingWildcard, TestSubQueryTransformerDistrib, MetricsHandlerTest, TestZkChroot, ZkNodePropsTest, NodeLostTriggerTest, TestFieldSortValues, HighlighterMaxOffsetTest, TestLMDirichletSimilarityFactory, LeaderElectionTest, TestSolrDeletionPolicy2, PKIAuthenticationIntegrationTest, TestReplicationHandlerBackup, CSVRequestHandlerTest, TestQueryTypes, HdfsDirectoryTest, TestDynamicFieldCollectionResource, TestClassicSimilarityFactory, ConfigSetsAPITest, XsltUpdateRequestHandlerTest, TestSolrCoreProperties, AlternateDirectoryTest, SmileWriterTest, FieldMutatingUpdateProcessorTest, TestDistributedGrouping, TestChildDocTransformer, EchoParamsTest, UnloadDistributedZkTest, IndexBasedSpellCheckerTest, RecoveryAfterSoftCommitTest, TestSearchPerf, OverseerTest, QueryElevationComponentTest, HLLSerializationTest, TestSolrCloudWithSecureImpersonation, SyncSliceTest]
   [junit4] Completed [522/764 (1!)] on J2 in 635.68s, 1 test, 1 error <<< FAILURES!

[...truncated 41685 lines...]

[JENKINS] Lucene-Solr-7.x-Linux (32bit/jdk1.8.0_144) - Build # 1109 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1109/
Java: 32bit/jdk1.8.0_144 -server -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.TestHdfsCloudBackupRestore.test

Error Message:
Error from server at https://127.0.0.1:42387/solr: Timed out waiting to see all replicas: [hdfsbackuprestore_restored_shard2_replica_n85] in cluster state.

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:42387/solr: Timed out waiting to see all replicas: [hdfsbackuprestore_restored_shard2_replica_n85] in cluster state.
	at __randomizedtesting.SeedInfo.seed([6138D22396EA590:8E47B2F89792C868]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:285)
	at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:140)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13469 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestHdfsCloudBackupRestore
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/init-core-data-001
   [junit4]   2> 2043463 WARN  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
   [junit4]   2> 2043463 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2043464 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2043465 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 2043527 WARN  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 2043546 WARN  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2043547 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2043565 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_localdomain_34411_hdfs____.lfptxv/webapp
   [junit4]   2> 2044140 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:34411
   [junit4]   2> 2044344 WARN  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2044345 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2044360 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_46159_datanode____.r177wm/webapp
   [junit4]   2> 2044932 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46159
   [junit4]   2> 2044972 WARN  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2044973 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2044985 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_34901_datanode____ygklsw/webapp
   [junit4]   2> 2045467 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34901
   [junit4]   2> 2045798 ERROR (DataNode: [[[DISK]file:/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to localhost.localdomain/127.0.0.1:45241) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 2045798 ERROR (DataNode: [[[DISK]file:/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to localhost.localdomain/127.0.0.1:45241) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 2045808 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x98a3db8dfaabc: from storage DS-4200e4f9-695a-4b1f-b29d-bc6dbfe18108 node DatanodeRegistration(127.0.0.1:33299, datanodeUuid=7141e948-b472-4b33-897d-47546a51efc3, infoPort=41515, infoSecurePort=0, ipcPort=39413, storageInfo=lv=-56;cid=testClusterID;nsid=1585893897;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 2045808 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x98a3db8dfaabc: from storage DS-6b9b635f-0914-427c-b9df-f757063022c6 node DatanodeRegistration(127.0.0.1:33299, datanodeUuid=7141e948-b472-4b33-897d-47546a51efc3, infoPort=41515, infoSecurePort=0, ipcPort=39413, storageInfo=lv=-56;cid=testClusterID;nsid=1585893897;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 2045814 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x98a3db8dfaac0: from storage DS-5a40da13-e3d4-4016-8ff2-6ae25cd6e227 node DatanodeRegistration(127.0.0.1:33213, datanodeUuid=73773d30-d3a1-4751-a1e4-b2da7ff9c87c, infoPort=45915, infoSecurePort=0, ipcPort=35093, storageInfo=lv=-56;cid=testClusterID;nsid=1585893897;c=0), blocks: 0, hasStaleStorage: true, processing time: 4 msecs
   [junit4]   2> 2045814 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x98a3db8dfaac0: from storage DS-5b7e3ffa-158b-4a3f-b404-426abae2e0c6 node DatanodeRegistration(127.0.0.1:33213, datanodeUuid=73773d30-d3a1-4751-a1e4-b2da7ff9c87c, infoPort=45915, infoSecurePort=0, ipcPort=35093, storageInfo=lv=-56;cid=testClusterID;nsid=1585893897;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 2045899 WARN  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.s.c.TestHdfsCloudBackupRestore The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 2050913 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002
   [junit4]   2> 2050914 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2050914 INFO  (Thread-3840) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2050914 INFO  (Thread-3840) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2050915 ERROR (Thread-3840) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2051014 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:38513
   [junit4]   2> 2051016 INFO  (zkConnectionManagerCallback-3524-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2051020 INFO  (jetty-launcher-3521-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2051020 INFO  (jetty-launcher-3521-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2051020 INFO  (jetty-launcher-3521-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1d2795e{/solr,null,AVAILABLE}
   [junit4]   2> 2051021 INFO  (jetty-launcher-3521-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@94ec64{/solr,null,AVAILABLE}
   [junit4]   2> 2051021 INFO  (jetty-launcher-3521-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@14b0f88{SSL,[ssl, http/1.1]}{127.0.0.1:42387}
   [junit4]   2> 2051021 INFO  (jetty-launcher-3521-thread-1) [    ] o.e.j.s.Server Started @2053057ms
   [junit4]   2> 2051021 INFO  (jetty-launcher-3521-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42387}
   [junit4]   2> 2051021 ERROR (jetty-launcher-3521-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2051021 INFO  (jetty-launcher-3521-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@8c21d9{SSL,[ssl, http/1.1]}{127.0.0.1:41963}
   [junit4]   2> 2051021 INFO  (jetty-launcher-3521-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 2051021 INFO  (jetty-launcher-3521-thread-2) [    ] o.e.j.s.Server Started @2053057ms
   [junit4]   2> 2051021 INFO  (jetty-launcher-3521-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2051021 INFO  (jetty-launcher-3521-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41963}
   [junit4]   2> 2051021 INFO  (jetty-launcher-3521-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2051021 INFO  (jetty-launcher-3521-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-03T12:55:35.731Z
   [junit4]   2> 2051021 ERROR (jetty-launcher-3521-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2051022 INFO  (jetty-launcher-3521-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 2051022 INFO  (jetty-launcher-3521-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2051022 INFO  (jetty-launcher-3521-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2051022 INFO  (jetty-launcher-3521-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-03T12:55:35.732Z
   [junit4]   2> 2051023 INFO  (zkConnectionManagerCallback-3526-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2051023 INFO  (jetty-launcher-3521-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2051025 INFO  (zkConnectionManagerCallback-3528-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2051026 INFO  (jetty-launcher-3521-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2051026 INFO  (jetty-launcher-3521-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1ecd9f3, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2051028 INFO  (jetty-launcher-3521-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1ecd9f3, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2051029 INFO  (jetty-launcher-3521-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38513/solr
   [junit4]   2> 2051030 INFO  (zkConnectionManagerCallback-3532-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2051030 INFO  (jetty-launcher-3521-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38513/solr
   [junit4]   2> 2051031 INFO  (zkConnectionManagerCallback-3536-thread-1-processing-n:127.0.0.1:42387_solr) [n:127.0.0.1:42387_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2051031 INFO  (zkConnectionManagerCallback-3538-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2051032 INFO  (zkConnectionManagerCallback-3540-thread-1-processing-n:127.0.0.1:41963_solr) [n:127.0.0.1:41963_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2051058 INFO  (jetty-launcher-3521-thread-1) [n:127.0.0.1:42387_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2051058 INFO  (jetty-launcher-3521-thread-2) [n:127.0.0.1:41963_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2051058 INFO  (jetty-launcher-3521-thread-1) [n:127.0.0.1:42387_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42387_solr
   [junit4]   2> 2051058 INFO  (jetty-launcher-3521-thread-2) [n:127.0.0.1:41963_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41963_solr
   [junit4]   2> 2051058 INFO  (jetty-launcher-3521-thread-1) [n:127.0.0.1:42387_solr    ] o.a.s.c.Overseer Overseer (id=99286000312254468-127.0.0.1:42387_solr-n_0000000000) starting
   [junit4]   2> 2051059 INFO  (zkCallback-3539-thread-1-processing-n:127.0.0.1:41963_solr) [n:127.0.0.1:41963_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2051059 INFO  (zkCallback-3535-thread-1-processing-n:127.0.0.1:42387_solr) [n:127.0.0.1:42387_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2051062 INFO  (jetty-launcher-3521-thread-1) [n:127.0.0.1:42387_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42387_solr
   [junit4]   2> 2051062 INFO  (zkCallback-3539-thread-1-processing-n:127.0.0.1:41963_solr) [n:127.0.0.1:41963_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2051069 INFO  (zkCallback-3535-thread-1-processing-n:127.0.0.1:42387_solr) [n:127.0.0.1:42387_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2051137 INFO  (jetty-launcher-3521-thread-2) [n:127.0.0.1:41963_solr    ] o.a.s.c.b.r.BackupRepositoryFactory Added backup repository with configuration params {type = repository,name = hdfs,class = org.apache.solr.core.backup.repository.HdfsBackupRepository,attributes = {name=hdfs, class=org.apache.solr.core.backup.repository.HdfsBackupRepository},args = {location=/backup,solr.hdfs.home=hdfs://localhost.localdomain:45241/solr,solr.hdfs.confdir=}}
   [junit4]   2> 2051137 INFO  (jetty-launcher-3521-thread-2) [n:127.0.0.1:41963_solr    ] o.a.s.c.b.r.BackupRepositoryFactory Default configuration for backup repository is with configuration params {type = repository,name = hdfs,class = org.apache.solr.core.backup.repository.HdfsBackupRepository,attributes = {name=hdfs, class=org.apache.solr.core.backup.repository.HdfsBackupRepository},args = {location=/backup,solr.hdfs.home=hdfs://localhost.localdomain:45241/solr,solr.hdfs.confdir=}}
   [junit4]   2> 2051142 INFO  (jetty-launcher-3521-thread-1) [n:127.0.0.1:42387_solr    ] o.a.s.c.b.r.BackupRepositoryFactory Added backup repository with configuration params {type = repository,name = hdfs,class = org.apache.solr.core.backup.repository.HdfsBackupRepository,attributes = {name=hdfs, class=org.apache.solr.core.backup.repository.HdfsBackupRepository},args = {location=/backup,solr.hdfs.home=hdfs://localhost.localdomain:45241/solr,solr.hdfs.confdir=}}
   [junit4]   2> 2051142 INFO  (jetty-launcher-3521-thread-1) [n:127.0.0.1:42387_solr    ] o.a.s.c.b.r.BackupRepositoryFactory Default configuration for backup repository is with configuration params {type = repository,name = hdfs,class = org.apache.solr.core.backup.repository.HdfsBackupRepository,attributes = {name=hdfs, class=org.apache.solr.core.backup.repository.HdfsBackupRepository},args = {location=/backup,solr.hdfs.home=hdfs://localhost.localdomain:45241/solr,solr.hdfs.confdir=}}
   [junit4]   2> 2051158 INFO  (jetty-launcher-3521-thread-2) [n:127.0.0.1:41963_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ecd9f3
   [junit4]   2> 2051159 INFO  (jetty-launcher-3521-thread-1) [n:127.0.0.1:42387_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ecd9f3
   [junit4]   2> 2051196 INFO  (jetty-launcher-3521-thread-2) [n:127.0.0.1:41963_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ecd9f3
   [junit4]   2> 2051196 INFO  (jetty-launcher-3521-thread-2) [n:127.0.0.1:41963_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ecd9f3
   [junit4]   2> 2051197 INFO  (jetty-launcher-3521-thread-1) [n:127.0.0.1:42387_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ecd9f3
   [junit4]   2> 2051197 INFO  (jetty-launcher-3521-thread-1) [n:127.0.0.1:42387_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ecd9f3
   [junit4]   2> 2051197 INFO  (jetty-launcher-3521-thread-2) [n:127.0.0.1:41963_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002/node2/.
   [junit4]   2> 2051198 INFO  (jetty-launcher-3521-thread-1) [n:127.0.0.1:42387_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002/node1/.
   [junit4]   2> 2051253 INFO  (zkConnectionManagerCallback-3546-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2051254 INFO  (zkConnectionManagerCallback-3550-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2051255 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2051255 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38513/solr ready
   [junit4]   2> 2051293 INFO  (TEST-TestHdfsCloudBackupRestore.test-seed#[6138D22396EA590]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 2051317 INFO  (qtp14533764-16052) [n:127.0.0.1:42387_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf1&version=2&pullReplicas=1&property.customKey=customValue&maxShardsPerNode=3&autoAddReplicas=true&name=hdfsbackuprestore&nrtReplicas=2&action=CREATE&numShards=2&tlogReplicas=0&wt=javabin and sendToOCPQueue=true
   [junit4]   2> 2051322 INFO  (OverseerThreadFactory-6221-thread-1-processing-n:127.0.0.1:42387_solr) [n:127.0.0.1:42387_solr    ] o.a.s.c.CreateCollectionCmd Create collection hdfsbackuprestore
   [junit4]   2> 2051322 WARN  (OverseerThreadFactory-6221-thread-1-processing-n:127.0.0.1:42387_solr) [n:127.0.0.1:42387_solr    ] o.a.s.c.CreateCollectionCmd Specified number of replicas of 3 on collection hdfsbackuprestore is higher than the number of Solr instances currently live or live and part of your createNodeSet(2). It's unusual to run two replica of the same slice on the same Solr-instance.
   [junit4]   2> 2051426 INFO  (OverseerStateUpdate-99286000312254468-127.0.0.1:42387_solr-n_0000000000) [n:127.0.0.1:42387_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"hdfsbackuprestore",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"hdfsbackuprestore_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:42387/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2051426 INFO  (OverseerStateUpdate-99286000312254468-127.0.0.1:42387_solr-n_0000000000) [n:127.0.0.1:42387_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"hdfsbackuprestore",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"hdfsbackuprestore_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:41963/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2051427 INFO  (OverseerStateUpdate-99286000312254468-127.0.0.1:42387_solr-n_0000000000) [n:127.0.0.1:42387_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"hdfsbackuprestore",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"hdfsbackuprestore_shard1_replica_p4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:42387/solr",
   [junit4]   2>   "type":"PULL",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2051428 INFO  (OverseerStateUpdate-99286000312254468-127.0.0.1:42387_solr-n_0000000000) [n:127.0.0.1:42387_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"hdfsbackuprestore",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"hdfsbackuprestore_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:41963/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2051428 INFO  (OverseerStateUpdate-99286000312254468-127.0.0.1:42387_solr-n_0000000000) [n:127.0.0.1:42387_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"hdfsbackuprestore",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"hdfsbackuprestore_shard2_replica_n8",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:42387/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2051429 INFO  (OverseerStateUpdate-99286000312254468-127.0.0.1:42387_solr-n_0000000000) [n:127.0.0.1:42387_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"hdfsbackuprestore",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"hdfsbackuprestore_shard2_replica_p10",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:41963/solr",
   [junit4]   2>   "type":"PULL",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2051637 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=hdfsbackuprestore&version=2&replicaType=PULL&property.customKey=customValue&coreNodeName=core_node7&name=hdfsbackuprestore_shard1_replica_p4&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 2051637 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2051641 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=hdfsbackuprestore&version=2&replicaType=NRT&property.customKey=customValue&coreNodeName=core_node11&name=hdfsbackuprestore_shard2_replica_n8&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 2051641 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=hdfsbackuprestore&version=2&replicaType=NRT&property.customKey=customValue&coreNodeName=core_node3&name=hdfsbackuprestore_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 2051665 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=hdfsbackuprestore&version=2&replicaType=NRT&property.customKey=customValue&coreNodeName=core_node5&name=hdfsbackuprestore_shard1_replica_n2&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 2051665 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2051674 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=hdfsbackuprestore&version=2&replicaType=PULL&property.customKey=customValue&coreNodeName=core_node12&name=hdfsbackuprestore_shard2_replica_p10&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 2051680 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=hdfsbackuprestore&version=2&replicaType=NRT&property.customKey=customValue&coreNodeName=core_node9&name=hdfsbackuprestore_shard2_replica_n6&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 2051782 INFO  (zkCallback-3535-thread-1-processing-n:127.0.0.1:42387_solr) [n:127.0.0.1:42387_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/hdfsbackuprestore/state.json] for collection [hdfsbackuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2051782 INFO  (zkCallback-3539-thread-1-processing-n:127.0.0.1:41963_solr) [n:127.0.0.1:41963_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/hdfsbackuprestore/state.json] for collection [hdfsbackuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2051782 INFO  (zkCallback-3539-thread-2-processing-n:127.0.0.1:41963_solr) [n:127.0.0.1:41963_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/hdfsbackuprestore/state.json] for collection [hdfsbackuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2052648 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 2052649 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 2052653 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.s.IndexSchema [hdfsbackuprestore_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 2052653 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.s.IndexSchema [hdfsbackuprestore_shard2_replica_n8] Schema name=minimal
   [junit4]   2> 2052654 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 2052655 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2052655 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2052655 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.CoreContainer Creating SolrCore 'hdfsbackuprestore_shard2_replica_n8' using configuration from collection hdfsbackuprestore, trusted=true
   [junit4]   2> 2052655 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'hdfsbackuprestore_shard1_replica_n1' using configuration from collection hdfsbackuprestore, trusted=true
   [junit4]   2> 2052655 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.hdfsbackuprestore.shard1.replica_n1' (registry 'solr.core.hdfsbackuprestore.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ecd9f3
   [junit4]   2> 2052655 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.hdfsbackuprestore.shard2.replica_n8' (registry 'solr.core.hdfsbackuprestore.shard2.replica_n8') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ecd9f3
   [junit4]   2> 2052655 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2052655 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2052655 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.SolrCore [[hdfsbackuprestore_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002/node1/hdfsbackuprestore_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002/node1/./hdfsbackuprestore_shard1_replica_n1/data/]
   [junit4]   2> 2052655 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.SolrCore [[hdfsbackuprestore_shard2_replica_n8] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002/node1/hdfsbackuprestore_shard2_replica_n8], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002/node1/./hdfsbackuprestore_shard2_replica_n8/data/]
   [junit4]   2> 2052659 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.s.IndexSchema [hdfsbackuprestore_shard1_replica_p4] Schema name=minimal
   [junit4]   2> 2052661 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2052661 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.c.CoreContainer Creating SolrCore 'hdfsbackuprestore_shard1_replica_p4' using configuration from collection hdfsbackuprestore, trusted=true
   [junit4]   2> 2052661 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.hdfsbackuprestore.shard1.replica_p4' (registry 'solr.core.hdfsbackuprestore.shard1.replica_p4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ecd9f3
   [junit4]   2> 2052661 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2052662 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.c.SolrCore [[hdfsbackuprestore_shard1_replica_p4] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002/node1/hdfsbackuprestore_shard1_replica_p4], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002/node1/./hdfsbackuprestore_shard1_replica_p4/data/]
   [junit4]   2> 2052685 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 2052688 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 2052695 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 2052696 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.s.IndexSchema [hdfsbackuprestore_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 2052699 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2052699 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'hdfsbackuprestore_shard1_replica_n2' using configuration from collection hdfsbackuprestore, trusted=true
   [junit4]   2> 2052699 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.hdfsbackuprestore.shard1.replica_n2' (registry 'solr.core.hdfsbackuprestore.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ecd9f3
   [junit4]   2> 2052699 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2052700 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.c.SolrCore [[hdfsbackuprestore_shard1_replica_n2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002/node2/hdfsbackuprestore_shard1_replica_n2], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002/node2/./hdfsbackuprestore_shard1_replica_n2/data/]
   [junit4]   2> 2052703 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.s.IndexSchema [hdfsbackuprestore_shard2_replica_n6] Schema name=minimal
   [junit4]   2> 2052719 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2052719 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'hdfsbackuprestore_shard2_replica_n6' using configuration from collection hdfsbackuprestore, trusted=true
   [junit4]   2> 2052719 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.s.IndexSchema [hdfsbackuprestore_shard2_replica_p10] Schema name=minimal
   [junit4]   2> 2052720 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.hdfsbackuprestore.shard2.replica_n6' (registry 'solr.core.hdfsbackuprestore.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ecd9f3
   [junit4]   2> 2052720 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2052720 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.c.SolrCore [[hdfsbackuprestore_shard2_replica_n6] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002/node2/hdfsbackuprestore_shard2_replica_n6], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002/node2/./hdfsbackuprestore_shard2_replica_n6/data/]
   [junit4]   2> 2052721 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2052721 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.c.CoreContainer Creating SolrCore 'hdfsbackuprestore_shard2_replica_p10' using configuration from collection hdfsbackuprestore, trusted=true
   [junit4]   2> 2052722 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.hdfsbackuprestore.shard2.replica_p10' (registry 'solr.core.hdfsbackuprestore.shard2.replica_p10') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ecd9f3
   [junit4]   2> 2052722 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2052722 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.c.SolrCore [[hdfsbackuprestore_shard2_replica_p10] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002/node2/hdfsbackuprestore_shard2_replica_p10], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001/tempDir-002/node2/./hdfsbackuprestore_shard2_replica_p10/data/]
   [junit4]   2> 2052728 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2052728 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2052729 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2052729 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2052732 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2052732 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@c116f2[hdfsbackuprestore_shard1_replica_n1] main]
   [junit4]   2> 2052732 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2052733 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2052733 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2052734 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2052734 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2052735 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2052735 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588576006905528320
   [junit4]   2> 2052735 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.s.SolrIndexSearcher Opening [Searcher@ebad4a[hdfsbackuprestore_shard2_replica_n8] main]
   [junit4]   2> 2052736 INFO  (searcherExecutor-6227-thread-1-processing-n:127.0.0.1:42387_solr x:hdfsbackuprestore_shard1_replica_n1 s:shard1 c:hdfsbackuprestore r:core_node3) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.SolrCore [hdfsbackuprestore_shard1_replica_n1] Registered new searcher Searcher@c116f2[hdfsbackuprestore_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2052737 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2052737 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2052739 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.s.SolrIndexSearcher Opening [Searcher@572fef[hdfsbackuprestore_shard1_replica_p4] main]
   [junit4]   2> 2052740 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2052740 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2052740 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2052740 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2052741 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2052741 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2052741 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.c.ZkController hdfsbackuprestore_shard1_replica_p4 starting background replication from leader
   [junit4]   2> 2052741 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
   [junit4]   2> 2052742 INFO  (searcherExecutor-6226-thread-1-processing-n:127.0.0.1:42387_solr x:hdfsbackuprestore_shard2_replica_n8 s:shard2 c:hdfsbackuprestore r:core_node11) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.SolrCore [hdfsbackuprestore_shard2_replica_n8] Registered new searcher Searcher@ebad4a[hdfsbackuprestore_shard2_replica_n8] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2052742 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 2052742 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588576006912868352
   [junit4]   2> 2052742 INFO  (searcherExecutor-6228-thread-1-processing-n:127.0.0.1:42387_solr x:hdfsbackuprestore_shard1_replica_p4 s:shard1 c:hdfsbackuprestore r:core_node7) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.c.SolrCore [hdfsbackuprestore_shard1_replica_p4] Registered new searcher Searcher@572fef[hdfsbackuprestore_shard1_replica_p4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2052743 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
   [junit4]   2> 2052743 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2052745 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 2052779 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2052779 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2052780 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2052780 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2052782 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@9685cb[hdfsbackuprestore_shard1_replica_n2] main]
   [junit4]   2> 2052783 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2052783 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2052783 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2052783 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2052783 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2052784 INFO  (searcherExecutor-6230-thread-1-processing-n:127.0.0.1:41963_solr x:hdfsbackuprestore_shard1_replica_n2 s:shard1 c:hdfsbackuprestore r:core_node5) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.c.SolrCore [hdfsbackuprestore_shard1_replica_n2] Registered new searcher Searcher@9685cb[hdfsbackuprestore_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2052784 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588576006956908544
   [junit4]   2> 2052785 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.s.SolrIndexSearcher Opening [Searcher@171bb4d[hdfsbackuprestore_shard2_replica_p10] main]
   [junit4]   2> 2052785 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2052786 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2052786 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2052787 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.c.ZkController hdfsbackuprestore_shard2_replica_p10 starting background replication from leader
   [junit4]   2> 2052787 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
   [junit4]   2> 2052787 INFO  (searcherExecutor-6233-thread-1-processing-n:127.0.0.1:41963_solr x:hdfsbackuprestore_shard2_replica_p10 s:shard2 c:hdfsbackuprestore r:core_node12) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.c.SolrCore [hdfsbackuprestore_shard2_replica_p10] Registered new searcher Searcher@171bb4d[hdfsbackuprestore_shard2_replica_p10] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2052787 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
   [junit4]   2> 2052787 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2052791 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2052792 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2052793 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2052793 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2052795 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@f783c4[hdfsbackuprestore_shard2_replica_n6] main]
   [junit4]   2> 2052796 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2052796 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2052797 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2052797 INFO  (searcherExecutor-6232-thread-1-processing-n:127.0.0.1:41963_solr x:hdfsbackuprestore_shard2_replica_n6 s:shard2 c:hdfsbackuprestore r:core_node9) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.c.SolrCore [hdfsbackuprestore_shard2_replica_n6] Registered new searcher Searcher@f783c4[hdfsbackuprestore_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2052797 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588576006970540032
   [junit4]   2> 2052846 INFO  (zkCallback-3539-thread-2-processing-n:127.0.0.1:41963_solr) [n:127.0.0.1:41963_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/hdfsbackuprestore/state.json] for collection [hdfsbackuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2052846 INFO  (zkCallback-3535-thread-2-processing-n:127.0.0.1:42387_solr) [n:127.0.0.1:42387_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/hdfsbackuprestore/state.json] for collection [hdfsbackuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2052846 INFO  (zkCallback-3539-thread-1-processing-n:127.0.0.1:41963_solr) [n:127.0.0.1:41963_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/hdfsbackuprestore/state.json] for collection [hdfsbackuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2053242 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2053242 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2053242 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:42387/solr/hdfsbackuprestore_shard1_replica_n1/
   [junit4]   2> 2053243 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=hdfsbackuprestore_shard1_replica_n1 url=https://127.0.0.1:42387/solr START replicas=[https://127.0.0.1:41963/solr/hdfsbackuprestore_shard1_replica_n2/] nUpdates=100
   [junit4]   2> 2053246 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2053246 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2053246 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:42387/solr/hdfsbackuprestore_shard2_replica_n8/
   [junit4]   2> 2053246 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.u.PeerSync PeerSync: core=hdfsbackuprestore_shard2_replica_n8 url=https://127.0.0.1:42387/solr START replicas=[https://127.0.0.1:41963/solr/hdfsbackuprestore_shard2_replica_n6/] nUpdates=100
   [junit4]   2> 2053250 INFO  (qtp32146622-16138) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.c.S.Request [hdfsbackuprestore_shard1_replica_n2]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2053252 INFO  (qtp32146622-16057) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.c.S.Request [hdfsbackuprestore_shard2_replica_n6]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2053543 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=hdfsbackuprestore_shard1_replica_n1 url=https://127.0.0.1:42387/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 2053543 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 2053543 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 2053543 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2053545 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:42387/solr/hdfsbackuprestore_shard1_replica_n1/ shard1
   [junit4]   2> 2053549 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.u.PeerSync PeerSync: core=hdfsbackuprestore_shard2_replica_n8 url=https://127.0.0.1:42387/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 2053549 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 2053549 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 2053549 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2053550 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:42387/solr/hdfsbackuprestore_shard2_replica_n8/ shard2
   [junit4]   2> 2053651 INFO  (zkCallback-3539-thread-2-processing-n:127.0.0.1:41963_solr) [n:127.0.0.1:41963_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/hdfsbackuprestore/state.json] for collection [hdfsbackuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2053651 INFO  (zkCallback-3539-thread-1-processing-n:127.0.0.1:41963_solr) [n:127.0.0.1:41963_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/hdfsbackuprestore/state.json] for collection [hdfsbackuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2053651 INFO  (zkCallback-3535-thread-2-processing-n:127.0.0.1:42387_solr) [n:127.0.0.1:42387_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/hdfsbackuprestore/state.json] for collection [hdfsbackuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2053689 INFO  (indexFetcher-6254-thread-1) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.h.IndexFetcher Replica core_node11 is leader but it's state is down, skipping replication
   [junit4]   2> 2053694 INFO  (indexFetcher-6244-thread-1) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.h.IndexFetcher Replica core_node3 is leader but it's state is down, skipping replication
   [junit4]   2> 2053695 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2053696 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=hdfsbackuprestore&version=2&replicaType=NRT&property.customKey=customValue&coreNodeName=core_node3&name=hdfsbackuprestore_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=0 QTime=2055
   [junit4]   2> 2053701 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2053702 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=hdfsbackuprestore&version=2&replicaType=NRT&property.customKey=customValue&coreNodeName=core_node11&name=hdfsbackuprestore_shard2_replica_n8&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=0 QTime=2060
   [junit4]   2> 2053744 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=hdfsbackuprestore&version=2&replicaType=PULL&property.customKey=customValue&coreNodeName=core_node7&name=hdfsbackuprestore_shard1_replica_p4&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=0 QTime=2107
   [junit4]   2> 2053788 INFO  (qtp32146622-16116) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=hdfsbackuprestore&version=2&replicaType=NRT&property.customKey=customValue&coreNodeName=core_node5&name=hdfsbackuprestore_shard1_replica_n2&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=0 QTime=2122
   [junit4]   2> 2053788 INFO  (qtp32146622-16055) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node12 x:hdfsbackuprestore_shard2_replica_p10] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=hdfsbackuprestore&version=2&replicaType=PULL&property.customKey=customValue&coreNodeName=core_node12&name=hdfsbackuprestore_shard2_replica_p10&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=0 QTime=2114
   [junit4]   2> 2053801 INFO  (qtp32146622-16045) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&collection=hdfsbackuprestore&version=2&replicaType=NRT&property.customKey=customValue&coreNodeName=core_node9&name=hdfsbackuprestore_shard2_replica_n6&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=0 QTime=2121
   [junit4]   2> 2053803 INFO  (qtp14533764-16052) [n:127.0.0.1:42387_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2053902 INFO  (zkCallback-3535-thread-2-processing-n:127.0.0.1:42387_solr) [n:127.0.0.1:42387_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/hdfsbackuprestore/state.json] for collection [hdfsbackuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2053902 INFO  (zkCallback-3539-thread-1-processing-n:127.0.0.1:41963_solr) [n:127.0.0.1:41963_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/hdfsbackuprestore/state.json] for collection [hdfsbackuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2053902 INFO  (zkCallback-3539-thread-2-processing-n:127.0.0.1:41963_solr) [n:127.0.0.1:41963_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/hdfsbackuprestore/state.json] for collection [hdfsbackuprestore] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2054804 INFO  (qtp14533764-16052) [n:127.0.0.1:42387_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&collection.configName=conf1&version=2&pullReplicas=1&property.customKey=customValue&maxShardsPerNode=3&autoAddReplicas=true&name=hdfsbackuprestore&nrtReplicas=2&action=CREATE&numShards=2&tlogReplicas=0&wt=javabin} status=0 QTime=3486
   [junit4]   2> 2054811 INFO  (qtp32146622-16049) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory [hdfsbackuprestore_shard2_replica_n6]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:42387/solr/hdfsbackuprestore_shard2_replica_n8/&wt=javabin&version=2}{add=[2 (1588576009076080640), 3 (1588576009080274944), 5 (1588576009080274945), 6 (1588576009080274946), 7 (1588576009080274947)]} 0 1
   [junit4]   2> 2054812 INFO  (qtp14533764-16054) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.u.p.LogUpdateProcessorFactory [hdfsbackuprestore_shard2_replica_n8]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2 (1588576009076080640), 3 (1588576009080274944), 5 (1588576009080274945), 6 (1588576009080274946), 7 (1588576009080274947)]} 0 6
   [junit4]   2> 2054820 INFO  (qtp32146622-16053) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [hdfsbackuprestore_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:42387/solr/hdfsbackuprestore_shard1_replica_n1/&wt=javabin&version=2}{add=[0 (1588576009078177792), 1 (1588576009080274944), 4 (1588576009080274945), 8 (1588576009080274946)]} 0 10
   [junit4]   2> 2054820 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [hdfsbackuprestore_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0 (1588576009078177792), 1 (1588576009080274944), 4 (1588576009080274945), 8 (1588576009080274946)]} 0 13
   [junit4]   2> 2054822 INFO  (qtp32146622-16057) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1588576009093906432,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2054822 INFO  (qtp32146622-16138) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1588576009093906432,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2054822 INFO  (qtp32146622-16057) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@1a51426 commitCommandVersion:1588576009093906432
   [junit4]   2> 2054822 INFO  (qtp32146622-16138) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@1a3451d commitCommandVersion:1588576009093906432
   [junit4]   2> 2054827 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1588576009099149312,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2054827 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@1fd9847 commitCommandVersion:1588576009099149312
   [junit4]   2> 2054829 INFO  (qtp14533764-16113) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1588576009101246464,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2054829 INFO  (qtp14533764-16113) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@65509e commitCommandVersion:1588576009101246464
   [junit4]   2> 2055039 INFO  (qtp32146622-16138) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@4d24cf[hdfsbackuprestore_shard2_replica_n6] main]
   [junit4]   2> 2055040 INFO  (qtp32146622-16138) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2055041 INFO  (searcherExecutor-6232-thread-1-processing-n:127.0.0.1:41963_solr x:hdfsbackuprestore_shard2_replica_n6 s:shard2 c:hdfsbackuprestore r:core_node9) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.c.SolrCore [hdfsbackuprestore_shard2_replica_n6] Registered new searcher Searcher@4d24cf[hdfsbackuprestore_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.3.0):C5)))}
   [junit4]   2> 2055041 INFO  (qtp32146622-16138) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory [hdfsbackuprestore_shard2_replica_n6]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:42387/solr/hdfsbackuprestore_shard2_replica_n8/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 219
   [junit4]   2> 2055043 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@ab3b2f[hdfsbackuprestore_shard1_replica_n1] main]
   [junit4]   2> 2055043 INFO  (qtp32146622-16057) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@1f2d90c[hdfsbackuprestore_shard1_replica_n2] main]
   [junit4]   2> 2055043 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2055043 INFO  (qtp32146622-16057) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2055044 INFO  (searcherExecutor-6230-thread-1-processing-n:127.0.0.1:41963_solr x:hdfsbackuprestore_shard1_replica_n2 s:shard1 c:hdfsbackuprestore r:core_node5) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.c.SolrCore [hdfsbackuprestore_shard1_replica_n2] Registered new searcher Searcher@1f2d90c[hdfsbackuprestore_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.3.0):C4)))}
   [junit4]   2> 2055044 INFO  (qtp32146622-16057) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard1 r:core_node5 x:hdfsbackuprestore_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [hdfsbackuprestore_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:42387/solr/hdfsbackuprestore_shard2_replica_n8/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 222
   [junit4]   2> 2055044 INFO  (searcherExecutor-6227-thread-1-processing-n:127.0.0.1:42387_solr x:hdfsbackuprestore_shard1_replica_n1 s:shard1 c:hdfsbackuprestore r:core_node3) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.SolrCore [hdfsbackuprestore_shard1_replica_n1] Registered new searcher Searcher@ab3b2f[hdfsbackuprestore_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.3.0):C4)))}
   [junit4]   2> 2055044 INFO  (qtp14533764-16048) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [hdfsbackuprestore_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:42387/solr/hdfsbackuprestore_shard2_replica_n8/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 216
   [junit4]   2> 2055047 INFO  (qtp14533764-16113) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.s.SolrIndexSearcher Opening [Searcher@eeeab3[hdfsbackuprestore_shard2_replica_n8] main]
   [junit4]   2> 2055048 INFO  (qtp14533764-16113) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2055048 INFO  (searcherExecutor-6226-thread-1-processing-n:127.0.0.1:42387_solr x:hdfsbackuprestore_shard2_replica_n8 s:shard2 c:hdfsbackuprestore r:core_node11) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.SolrCore [hdfsbackuprestore_shard2_replica_n8] Registered new searcher Searcher@eeeab3[hdfsbackuprestore_shard2_replica_n8] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.3.0):C5)))}
   [junit4]   2> 2055048 INFO  (qtp14533764-16113) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.u.p.LogUpdateProcessorFactory [hdfsbackuprestore_shard2_replica_n8]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:42387/solr/hdfsbackuprestore_shard2_replica_n8/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 219
   [junit4]   2> 2055049 INFO  (qtp14533764-16052) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.u.p.LogUpdateProcessorFactory [hdfsbackuprestore_shard2_replica_n8]  webapp=/solr path=/update params={_stateVer_=hdfsbackuprestore:5&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 227
   [junit4]   2> 2055050 INFO  (qtp14533764-16056) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.c.S.Request [hdfsbackuprestore_shard2_replica_n8]  webapp=/solr path=/select params={q=*:*&distrib=false&wt=javabin&version=2} hits=5 status=0 QTime=0
   [junit4]   2> 2055050 INFO  (qtp14533764-16042) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node3 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.c.S.Request [hdfsbackuprestore_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&distrib=false&wt=javabin&version=2} hits=4 status=0 QTime=0
   [junit4]   2> 2055051 INFO  (TEST-TestHdfsCloudBackupRestore.test-seed#[6138D22396EA590]) [    ] o.a.s.c.AbstractCloudBackupRestoreTestCase Triggering Backup command
   [junit4]   2> 2055051 INFO  (qtp14533764-16046) [n:127.0.0.1:42387_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :backup with params async=6b083a09-9c6d-432a-8cef-dd64fcab1cde&name=mytestbackup&action=BACKUP&collection=hdfsbackuprestore&repository=hdfs&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2055052 INFO  (qtp14533764-16046) [n:127.0.0.1:42387_solr    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost.localdomain:45241/solr
   [junit4]   2> 2055052 INFO  (qtp14533764-16046) [n:127.0.0.1:42387_solr    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 2055062 INFO  (qtp14533764-16046) [n:127.0.0.1:42387_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={async=6b083a09-9c6d-432a-8cef-dd64fcab1cde&name=mytestbackup&action=BACKUP&collection=hdfsbackuprestore&repository=hdfs&wt=javabin&version=2} status=0 QTime=11
   [junit4]   2> 2055063 INFO  (qtp14533764-16046) [n:127.0.0.1:42387_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=6b083a09-9c6d-432a-8cef-dd64fcab1cde&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2055063 INFO  (qtp14533764-16046) [n:127.0.0.1:42387_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=6b083a09-9c6d-432a-8cef-dd64fcab1cde&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2055066 INFO  (OverseerThreadFactory-6221-thread-2-processing-n:127.0.0.1:42387_solr) [n:127.0.0.1:42387_solr    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost.localdomain:45241/solr
   [junit4]   2> 2055066 INFO  (OverseerCollectionConfigSetProcessor-99286000312254468-127.0.0.1:42387_solr-n_0000000000) [n:127.0.0.1:42387_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/c

[...truncated too long message...]

   2> 2089922 INFO  (coreCloseExecutor-6270-thread-3) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@19198a4: rootName = null, domain = solr.core.hdfsbackuprestore.shard2.replica_n6, service url = null, agent id = null] for registry solr.core.hdfsbackuprestore.shard2.replica_n6 / com.codahale.metrics.MetricRegistry@19a9565
   [junit4]   2> 2089924 INFO  (coreCloseExecutor-6271-thread-4) [n:127.0.0.1:42387_solr c:hdfsbackuprestore_restored s:shard1 r:core_node84 x:hdfsbackuprestore_restored_shard1_replica_n83] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.hdfsbackuprestore_restored.shard1.replica_n83, tag=21588703
   [junit4]   2> 2089925 INFO  (coreCloseExecutor-6271-thread-4) [n:127.0.0.1:42387_solr c:hdfsbackuprestore_restored s:shard1 r:core_node84 x:hdfsbackuprestore_restored_shard1_replica_n83] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@111f3ca: rootName = null, domain = solr.core.hdfsbackuprestore_restored.shard1.replica_n83, service url = null, agent id = null] for registry solr.core.hdfsbackuprestore_restored.shard1.replica_n83 / com.codahale.metrics.MetricRegistry@4e24fc
   [junit4]   2> 2089928 INFO  (coreCloseExecutor-6270-thread-4) [n:127.0.0.1:41963_solr c:hdfsbackuprestore_restored s:shard2 r:core_node82 x:hdfsbackuprestore_restored_shard2_replica_n81] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.hdfsbackuprestore_restored.shard2.leader, tag=16599370
   [junit4]   2> 2089929 INFO  (coreCloseExecutor-6270-thread-3) [n:127.0.0.1:41963_solr c:hdfsbackuprestore s:shard2 r:core_node9 x:hdfsbackuprestore_shard2_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.hdfsbackuprestore.shard2.leader, tag=31018469
   [junit4]   2> 2089931 INFO  (jetty-closer-3522-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@94ec64{/solr,null,UNAVAILABLE}
   [junit4]   2> 2089932 INFO  (coreCloseExecutor-6271-thread-2) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard1 r:core_node7 x:hdfsbackuprestore_shard1_replica_p4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.hdfsbackuprestore.shard1.leader, tag=13695963
   [junit4]   2> 2089932 INFO  (coreCloseExecutor-6271-thread-3) [n:127.0.0.1:42387_solr c:hdfsbackuprestore s:shard2 r:core_node11 x:hdfsbackuprestore_shard2_replica_n8] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.hdfsbackuprestore.shard2.leader, tag=9274380
   [junit4]   2> 2089932 INFO  (coreCloseExecutor-6271-thread-4) [n:127.0.0.1:42387_solr c:hdfsbackuprestore_restored s:shard1 r:core_node84 x:hdfsbackuprestore_restored_shard1_replica_n83] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.hdfsbackuprestore_restored.shard1.leader, tag=21588703
   [junit4]   2> 2089934 INFO  (jetty-closer-3522-thread-1) [    ] o.a.s.c.Overseer Overseer (id=99286000312254468-127.0.0.1:42387_solr-n_0000000000) closing
   [junit4]   2> 2089934 INFO  (OverseerStateUpdate-99286000312254468-127.0.0.1:42387_solr-n_0000000000) [n:127.0.0.1:42387_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:42387_solr
   [junit4]   2> 2089934 WARN  (OverseerAutoScalingTriggerThread-99286000312254468-127.0.0.1:42387_solr-n_0000000000) [n:127.0.0.1:42387_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 2089936 INFO  (jetty-closer-3522-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1d2795e{/solr,null,UNAVAILABLE}
   [junit4]   2> 2089936 ERROR (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2089936 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[6138D22396EA590]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38513 38513
   [junit4]   2> 2089963 INFO  (Thread-3840) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38513 38513
   [junit4]   2> 2089964 WARN  (Thread-3840) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/aliases.json
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 	2	/solr/configs/conf1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	8	/solr/collections/hdfsbackuprestore_restored/state.json
   [junit4]   2> 	7	/solr/collections/hdfsbackuprestore/state.json
   [junit4]   2> 	3	/solr/clusterstate.json
   [junit4]   2> 	3	/solr/clusterprops.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 	3	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestHdfsCloudBackupRestore_6138D22396EA590-001
   [junit4]   2> Jan 03, 2018 12:56:14 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 68 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {shard_s=PostingsFormat(name=Direct), id=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{}, maxPointsInLeafNode=191, maxMBSortInHeap=6.440772782226201, sim=RandomSimilarity(queryNorm=false): {}, locale=es-CO, timezone=Pacific/Pago_Pago
   [junit4]   2> NOTE: Linux 4.10.0-40-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=2,free=181947176,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [MultiTermTest, TestSolrCloudWithHadoopAuthPlugin, TestRandomFaceting, ResponseLogComponentTest, TestOrdValues, DateFieldTest, SaslZkACLProviderTest, StatsReloadRaceTest, TestXmlQParser, TestObjectReleaseTracker, TestConfig, TestIBSimilarityFactory, CacheHeaderTest, OverseerTaskQueueTest, TestReqParamsAPI, CleanupOldIndexTest, TestFilteredDocIdSet, TestSolrCloudWithDelegationTokens, ConnectionManagerTest, ManagedSchemaRoundRobinCloudTest, TestSchemaResource, DistributedMLTComponentTest, TestStressVersions, TestRandomDVFaceting, ReplicaListTransformerTest, TestCodecSupport, UpdateParamsTest, DataDrivenBlockJoinTest, VMParamsZkACLAndCredentialsProvidersTest, MetricTriggerTest, TestManagedResource, TestConfigsApi, TestPHPSerializedResponseWriter, TestSystemCollAutoCreate, DistributedFacetPivotWhiteBoxTest, CachingDirectoryFactoryTest, DeleteReplicaTest, ClassificationUpdateProcessorTest, DistributedFacetPivotLongTailTest, TestQueryWrapperFilter, BasicDistributedZkTest, BasicAuthIntegrationTest, TestBulkSchemaAPI, BigEndianAscendingWordDeserializerTest, ForceLeaderTest, TestDistribStateManager, DistributedExpandComponentTest, HdfsChaosMonkeyNothingIsSafeTest, TestComponentsName, TestOnReconnectListenerSupport, TestFastOutputStream, TestSha256AuthenticationProvider, TestReRankQParserPlugin, TestPointFields, TestFieldCacheReopen, ChaosMonkeyNothingIsSafeWithPullReplicasTest, TestDistributedSearch, SolrMetricReporterTest, QueryResultKeyTest, TestPhraseSuggestions, SearchHandlerTest, TermVectorComponentTest, TestIntervalFaceting, TestBadConfig, TestAddFieldRealTimeGet, TestRebalanceLeaders, TriggerIntegrationTest, MissingSegmentRecoveryTest, PropertiesRequestHandlerTest, SimpleCollectionCreateDeleteTest, TestSolrJ, TestUtilizeNode, TestNumericTokenStream, TestInPlaceUpdatesDistrib, SimpleFacetsTest, URLClassifyProcessorTest, TestCloudPseudoReturnFields, DistribCursorPagingTest, DocumentBuilderTest, MetricsConfigTest, CollectionsAPISolrJTest, ExplicitHLLTest, TestLuceneIndexBackCompat, TestOverriddenPrefixQueryForCustomFieldType, PeerSyncTest, TestStandardQParsers, MinimalSchemaTest, TestHighlightDedupGrouping, ExecutePlanActionTest, AnalyticsQueryTest, LeaderInitiatedRecoveryOnShardRestartTest, ActionThrottleTest, SearchRateTriggerTest, TestSort, SoftAutoCommitTest, TestExactSharedStatsCache, TestFreeTextSuggestions, StressHdfsTest, TestRetrieveFieldsOptimizer, TestImplicitCoreProperties, TestMultiValuedNumericRangeQuery, TestBinaryField, TestCSVLoader, TestInitQParser, TimeRoutedAliasUpdateProcessorTest, TestSweetSpotSimilarityFactory, TestPostingsSolrHighlighter, TestFoldingMultitermQuery, BasicZkTest, UpdateRequestProcessorFactoryTest, DirectoryFactoryTest, TestPolicyCloud, TestStressCloudBlindAtomicUpdates, SolrSlf4jReporterTest, TestCloudInspectUtil, TestCloudSchemaless, SliceStateTest, TestMissingGroups, TestManagedSynonymFilterFactory, CdcrVersionReplicationTest, TestSolrCloudSnapshots, TestFieldTypeCollectionResource, SolrPluginUtilsTest, BooleanFieldTest, TestComplexPhraseQParserPlugin, TestConfigSetImmutable, TestValueSourceCache, FieldAnalysisRequestHandlerTest, QueryParsingTest, JsonLoaderTest, PingRequestHandlerTest, SolrTestCaseJ4Test, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, TestEmbeddedSolrServerConstructors, TestEmbeddedSolrServerSchemaAPI, ConnectionReuseTest, AliasIntegrationTest, AsyncCallRequestStatusResponseTest, ChaosMonkeySafeLeaderWithPullReplicasTest, DistribJoinFromCollectionTest, LeaderElectionContextKeyTest, LeaderFailoverAfterPartitionTest, MultiThreadedOCPTest, OverseerCollectionConfigSetProcessorTest, OverseerModifyCollectionTest, OverseerStatusTest, ReplaceNodeTest, SharedFSAutoReplicaFailoverTest, SolrCLIZkUtilsTest, SolrCloudExampleTest, TestClusterProperties, TestCollectionAPI, TestConfigSetsAPI, TestDownShardTolerantSearch, TestHdfsCloudBackupRestore]
   [junit4] Completed [554/764 (1!)] on J0 in 54.46s, 1 test, 1 error <<< FAILURES!

[...truncated 44653 lines...]

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-9.0.1) - Build # 1108 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1108/
Java: 64bit/jdk-9.0.1 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest.test

Error Message:
expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([E7B212BFFE4FF2E8:6FE62D6550B39F10]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest.assertInvariants(TimeRoutedAliasUpdateProcessorTest.java:245)
	at org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest.test(TimeRoutedAliasUpdateProcessorTest.java:123)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 11828 lines...]
   [junit4] Suite: org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest
   [junit4]   2> 185900 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E7B212BFFE4FF2E8]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/init-core-data-001
   [junit4]   2> 185901 WARN  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E7B212BFFE4FF2E8]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 185901 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E7B212BFFE4FF2E8]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 185902 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E7B212BFFE4FF2E8]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0)
   [junit4]   2> 185903 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E7B212BFFE4FF2E8]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001
   [junit4]   2> 185903 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E7B212BFFE4FF2E8]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 185903 INFO  (Thread-267) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 185903 INFO  (Thread-267) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 185912 ERROR (Thread-267) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 186003 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E7B212BFFE4FF2E8]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:42695
   [junit4]   2> 186027 INFO  (zkConnectionManagerCallback-247-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186035 INFO  (jetty-launcher-244-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 186054 INFO  (jetty-launcher-244-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 186157 INFO  (jetty-launcher-244-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@64a6309e{/solr,null,AVAILABLE}
   [junit4]   2> 186158 INFO  (jetty-launcher-244-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@d979554{SSL,[ssl, http/1.1]}{127.0.0.1:39429}
   [junit4]   2> 186158 INFO  (jetty-launcher-244-thread-1) [    ] o.e.j.s.Server Started @187774ms
   [junit4]   2> 186158 INFO  (jetty-launcher-244-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39429}
   [junit4]   2> 186159 ERROR (jetty-launcher-244-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 186159 INFO  (jetty-launcher-244-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 186159 INFO  (jetty-launcher-244-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 186159 INFO  (jetty-launcher-244-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 186159 INFO  (jetty-launcher-244-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-03T08:35:43.766309Z
   [junit4]   2> 186161 INFO  (jetty-launcher-244-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d273b7{/solr,null,AVAILABLE}
   [junit4]   2> 186172 INFO  (jetty-launcher-244-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1e6a125d{SSL,[ssl, http/1.1]}{127.0.0.1:36085}
   [junit4]   2> 186172 INFO  (jetty-launcher-244-thread-2) [    ] o.e.j.s.Server Started @187788ms
   [junit4]   2> 186172 INFO  (jetty-launcher-244-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36085}
   [junit4]   2> 186173 ERROR (jetty-launcher-244-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 186173 INFO  (jetty-launcher-244-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 186173 INFO  (jetty-launcher-244-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 186173 INFO  (jetty-launcher-244-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 186173 INFO  (jetty-launcher-244-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-03T08:35:43.780852Z
   [junit4]   2> 186174 INFO  (zkConnectionManagerCallback-249-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186195 INFO  (jetty-launcher-244-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 186224 INFO  (zkConnectionManagerCallback-251-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186225 INFO  (jetty-launcher-244-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 186225 INFO  (jetty-launcher-244-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42695/solr
   [junit4]   2> 186258 INFO  (jetty-launcher-244-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42695/solr
   [junit4]   2> 186273 INFO  (zkConnectionManagerCallback-255-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186326 INFO  (zkConnectionManagerCallback-261-thread-1-processing-n:127.0.0.1:39429_solr) [n:127.0.0.1:39429_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186327 INFO  (zkConnectionManagerCallback-259-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186354 INFO  (zkConnectionManagerCallback-264-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186360 INFO  (jetty-launcher-244-thread-2) [n:127.0.0.1:36085_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 186360 INFO  (jetty-launcher-244-thread-2) [n:127.0.0.1:36085_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36085_solr
   [junit4]   2> 186360 INFO  (jetty-launcher-244-thread-2) [n:127.0.0.1:36085_solr    ] o.a.s.c.Overseer Overseer (id=99284978467930118-127.0.0.1:36085_solr-n_0000000000) starting
   [junit4]   2> 186369 INFO  (jetty-launcher-244-thread-2) [n:127.0.0.1:36085_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36085_solr
   [junit4]   2> 186373 INFO  (zkCallback-263-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 186403 INFO  (jetty-launcher-244-thread-1) [n:127.0.0.1:39429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 186404 INFO  (jetty-launcher-244-thread-1) [n:127.0.0.1:39429_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 186422 INFO  (jetty-launcher-244-thread-1) [n:127.0.0.1:39429_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39429_solr
   [junit4]   2> 186437 INFO  (zkCallback-263-thread-2-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 186440 INFO  (zkCallback-260-thread-1-processing-n:127.0.0.1:39429_solr) [n:127.0.0.1:39429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 186442 INFO  (jetty-launcher-244-thread-2) [n:127.0.0.1:36085_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36085.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2063cea0
   [junit4]   2> 186458 INFO  (jetty-launcher-244-thread-2) [n:127.0.0.1:36085_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36085.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2063cea0
   [junit4]   2> 186458 INFO  (jetty-launcher-244-thread-2) [n:127.0.0.1:36085_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36085.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2063cea0
   [junit4]   2> 186459 INFO  (jetty-launcher-244-thread-2) [n:127.0.0.1:36085_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/.
   [junit4]   2> 186537 INFO  (jetty-launcher-244-thread-1) [n:127.0.0.1:39429_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39429.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2063cea0
   [junit4]   2> 186544 INFO  (jetty-launcher-244-thread-1) [n:127.0.0.1:39429_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39429.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2063cea0
   [junit4]   2> 186544 INFO  (jetty-launcher-244-thread-1) [n:127.0.0.1:39429_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39429.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2063cea0
   [junit4]   2> 186545 INFO  (jetty-launcher-244-thread-1) [n:127.0.0.1:39429_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node1/.
   [junit4]   2> 186580 INFO  (zkConnectionManagerCallback-269-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186590 INFO  (TEST-TimeRoutedAliasUpdateProcessorTest.testParse-seed#[E7B212BFFE4FF2E8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testParse
   [junit4]   2> 186591 INFO  (TEST-TimeRoutedAliasUpdateProcessorTest.testParse-seed#[E7B212BFFE4FF2E8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testParse
   [junit4]   2> 186619 INFO  (TEST-TimeRoutedAliasUpdateProcessorTest.test-seed#[E7B212BFFE4FF2E8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 186624 INFO  (zkConnectionManagerCallback-274-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186625 INFO  (TEST-TimeRoutedAliasUpdateProcessorTest.test-seed#[E7B212BFFE4FF2E8]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 186625 INFO  (TEST-TimeRoutedAliasUpdateProcessorTest.test-seed#[E7B212BFFE4FF2E8]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42695/solr ready
   [junit4]   2> 186682 INFO  (qtp2145550292-1409) [n:127.0.0.1:39429_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=1&name=timeConfig&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 186697 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.CreateCollectionCmd Create collection timeConfig
   [junit4]   2> 186703 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.CreateCollectionCmd Only _default config set found, using it.
   [junit4]   2> 186704 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/protwords.txt to /configs/timeConfig/protwords.txt
   [junit4]   2> 186706 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/managed-schema to /configs/timeConfig/managed-schema
   [junit4]   2> 186707 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/solrconfig.xml to /configs/timeConfig/solrconfig.xml
   [junit4]   2> 186709 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/synonyms.txt to /configs/timeConfig/synonyms.txt
   [junit4]   2> 186710 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/stopwords.txt to /configs/timeConfig/stopwords.txt
   [junit4]   2> 186712 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_it.txt to /configs/timeConfig/lang/contractions_it.txt
   [junit4]   2> 186714 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stemdict_nl.txt to /configs/timeConfig/lang/stemdict_nl.txt
   [junit4]   2> 186716 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hy.txt to /configs/timeConfig/lang/stopwords_hy.txt
   [junit4]   2> 186717 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_no.txt to /configs/timeConfig/lang/stopwords_no.txt
   [junit4]   2> 186717 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ca.txt to /configs/timeConfig/lang/contractions_ca.txt
   [junit4]   2> 186718 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_id.txt to /configs/timeConfig/lang/stopwords_id.txt
   [junit4]   2> 186719 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ga.txt to /configs/timeConfig/lang/stopwords_ga.txt
   [junit4]   2> 186720 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hi.txt to /configs/timeConfig/lang/stopwords_hi.txt
   [junit4]   2> 186721 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_da.txt to /configs/timeConfig/lang/stopwords_da.txt
   [junit4]   2> 186723 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ja.txt to /configs/timeConfig/lang/stopwords_ja.txt
   [junit4]   2> 186725 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_pt.txt to /configs/timeConfig/lang/stopwords_pt.txt
   [junit4]   2> 186729 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hu.txt to /configs/timeConfig/lang/stopwords_hu.txt
   [junit4]   2> 186729 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_el.txt to /configs/timeConfig/lang/stopwords_el.txt
   [junit4]   2> 186730 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ru.txt to /configs/timeConfig/lang/stopwords_ru.txt
   [junit4]   2> 186734 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_tr.txt to /configs/timeConfig/lang/stopwords_tr.txt
   [junit4]   2> 186735 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ar.txt to /configs/timeConfig/lang/stopwords_ar.txt
   [junit4]   2> 186737 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/userdict_ja.txt to /configs/timeConfig/lang/userdict_ja.txt
   [junit4]   2> 186738 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_eu.txt to /configs/timeConfig/lang/stopwords_eu.txt
   [junit4]   2> 186741 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_it.txt to /configs/timeConfig/lang/stopwords_it.txt
   [junit4]   2> 186743 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_cz.txt to /configs/timeConfig/lang/stopwords_cz.txt
   [junit4]   2> 186746 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stoptags_ja.txt to /configs/timeConfig/lang/stoptags_ja.txt
   [junit4]   2> 186748 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_fr.txt to /configs/timeConfig/lang/contractions_fr.txt
   [junit4]   2> 186751 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_de.txt to /configs/timeConfig/lang/stopwords_de.txt
   [junit4]   2> 186757 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fa.txt to /configs/timeConfig/lang/stopwords_fa.txt
   [junit4]   2> 186758 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ga.txt to /configs/timeConfig/lang/contractions_ga.txt
   [junit4]   2> 186759 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ca.txt to /configs/timeConfig/lang/stopwords_ca.txt
   [junit4]   2> 186761 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_nl.txt to /configs/timeConfig/lang/stopwords_nl.txt
   [junit4]   2> 186761 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_bg.txt to /configs/timeConfig/lang/stopwords_bg.txt
   [junit4]   2> 186762 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_lv.txt to /configs/timeConfig/lang/stopwords_lv.txt
   [junit4]   2> 186764 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_en.txt to /configs/timeConfig/lang/stopwords_en.txt
   [junit4]   2> 186765 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_es.txt to /configs/timeConfig/lang/stopwords_es.txt
   [junit4]   2> 186767 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_th.txt to /configs/timeConfig/lang/stopwords_th.txt
   [junit4]   2> 186769 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ro.txt to /configs/timeConfig/lang/stopwords_ro.txt
   [junit4]   2> 186772 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fr.txt to /configs/timeConfig/lang/stopwords_fr.txt
   [junit4]   2> 186773 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_sv.txt to /configs/timeConfig/lang/stopwords_sv.txt
   [junit4]   2> 186774 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fi.txt to /configs/timeConfig/lang/stopwords_fi.txt
   [junit4]   2> 186777 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/hyphenations_ga.txt to /configs/timeConfig/lang/hyphenations_ga.txt
   [junit4]   2> 186777 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_gl.txt to /configs/timeConfig/lang/stopwords_gl.txt
   [junit4]   2> 186778 INFO  (OverseerThreadFactory-788-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/params.json to /configs/timeConfig/params.json
   [junit4]   2> 186890 INFO  (OverseerStateUpdate-99284978467930118-127.0.0.1:36085_solr-n_0000000000) [n:127.0.0.1:36085_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"timeConfig",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"timeConfig_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:36085/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 187111 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=timeConfig&newCollection=true&name=timeConfig_shard1_replica_n1&action=CREATE&numShards=1&collection=timeConfig&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 187112 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 187216 INFO  (zkCallback-263-thread-1-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/timeConfig/state.json] for collection [timeConfig] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 187216 INFO  (zkCallback-263-thread-2-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/timeConfig/state.json] for collection [timeConfig] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 188125 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.RequestParams conf resource params.json loaded . version : 0 
   [junit4]   2> 188126 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 188126 WARN  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/extraction/lib
   [junit4]   2> 188127 WARN  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 188127 WARN  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/clustering/lib
   [junit4]   2> 188127 WARN  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 188127 WARN  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/langid/lib
   [junit4]   2> 188127 WARN  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 188127 WARN  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/velocity/lib
   [junit4]   2> 188127 WARN  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-velocity-\d.*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 188131 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 188159 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.IndexSchema [timeConfig_shard1_replica_n1] Schema name=default-config
   [junit4]   2> 188367 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 188389 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'timeConfig_shard1_replica_n1' using configuration from collection timeConfig, trusted=true
   [junit4]   2> 188390 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36085.solr.core.timeConfig.shard1.replica_n1' (registry 'solr.core.timeConfig.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2063cea0
   [junit4]   2> 188390 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 188390 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrCore [[timeConfig_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/./timeConfig_shard1_replica_n1/data/]
   [junit4]   2> 188409 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
   [junit4]   2> 188537 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 188537 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 188538 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 188538 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 188540 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3368815f[timeConfig_shard1_replica_n1] main]
   [junit4]   2> 188541 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/timeConfig
   [junit4]   2> 188541 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/timeConfig
   [junit4]   2> 188542 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/timeConfig/managed-schema
   [junit4]   2> 188543 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 188543 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 188546 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 188552 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 188553 INFO  (searcherExecutor-793-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@3368815f[timeConfig_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 188553 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588559658258268160
   [junit4]   2> 188553 INFO  (searcherExecutor-793-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener done.
   [junit4]   2> 188553 INFO  (searcherExecutor-793-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 188555 INFO  (searcherExecutor-793-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrCore [timeConfig_shard1_replica_n1] Registered new searcher Searcher@3368815f[timeConfig_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 188556 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 188556 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 188556 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:36085/solr/timeConfig_shard1_replica_n1/
   [junit4]   2> 188556 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 188556 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:36085/solr/timeConfig_shard1_replica_n1/ has no replicas
   [junit4]   2> 188556 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 188558 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:36085/solr/timeConfig_shard1_replica_n1/ shard1
   [junit4]   2> 188659 INFO  (zkCallback-263-thread-2-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/timeConfig/state.json] for collection [timeConfig] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 188659 INFO  (zkCallback-263-thread-3-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/timeConfig/state.json] for collection [timeConfig] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 188708 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 188757 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=timeConfig&newCollection=true&name=timeConfig_shard1_replica_n1&action=CREATE&numShards=1&collection=timeConfig&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1645
   [junit4]   2> 188759 INFO  (qtp2145550292-1409) [n:127.0.0.1:39429_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 188810 INFO  (zkCallback-263-thread-2-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/timeConfig/state.json] for collection [timeConfig] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 188810 INFO  (zkCallback-263-thread-3-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/timeConfig/state.json] for collection [timeConfig] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 189760 INFO  (qtp2145550292-1409) [n:127.0.0.1:39429_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=1&name=timeConfig&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2} status=0 QTime=3077
   [junit4]   2> 189773 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Executed config commands successfully and persisted to ZK [{"set-user-property":{"timePartitionAliasName":"myalias"}}, {"set-user-property":{"update.autoCreateFields":false}}, {"add-updateprocessor":{
   [junit4]   2>     "name":"tolerant",
   [junit4]   2>     "class":"solr.TolerantUpdateProcessorFactory"}}, {"add-updateprocessor":{
   [junit4]   2>     "name":"inc",
   [junit4]   2>     "class":"org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest$IncrementURPFactory",
   [junit4]   2>     "fieldName":"integer_i"}}]
   [junit4]   2> 189773 INFO  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.c.SolrCore config update listener called for core timeConfig_shard1_replica_n1
   [junit4]   2> 189773 INFO  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.c.SolrCore core reload timeConfig_shard1_replica_n1
   [junit4]   2> 189775 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Waiting up to 30 secs for 1 replicas to set the property overlay to be of version 0 for collection timeConfig
   [junit4]   2> 189776 INFO  (solrHandlerExecutor-798-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 189782 INFO  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.c.RequestParams conf resource params.json loaded . version : 0 
   [junit4]   2> 189782 INFO  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 189783 WARN  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/extraction/lib
   [junit4]   2> 189783 WARN  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 189783 WARN  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/clustering/lib
   [junit4]   2> 189783 WARN  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 189783 WARN  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/langid/lib
   [junit4]   2> 189783 WARN  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 189783 WARN  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../contrib/velocity/lib
   [junit4]   2> 189783 WARN  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-velocity-\d.*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 189787 INFO  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 189798 INFO  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.s.IndexSchema [timeConfig_shard1_replica_n1] Schema name=default-config
   [junit4]   2> 189884 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 189884 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.S.Request [timeConfig_shard1_replica_n1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 189884 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 189885 INFO  (solrHandlerExecutor-798-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from https://127.0.0.1:36085/solr/timeConfig_shard1_replica_n1/ for prop overlay   after 1 attempts
   [junit4]   2> 189885 INFO  (solrHandlerExecutor-798-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 189904 INFO  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 189918 INFO  (Thread-286) [n:127.0.0.1:36085_solr    ] o.a.s.c.CoreContainer Reloading SolrCore 'timeConfig_shard1_replica_n1' using configuration from collection timeConfig
   [junit4]   2> 189934 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36085.solr.core.timeConfig.shard1.replica_n1' (registry 'solr.core.timeConfig.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2063cea0
   [junit4]   2> 189934 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrCore [[timeConfig_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/timeConfig_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/./timeConfig_shard1_replica_n1/data/]
   [junit4]   2> 189936 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
   [junit4]   2> 189969 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 189969 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 189970 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@67d16598[timeConfig_shard1_replica_n1] main]
   [junit4]   2> 189971 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/timeConfig
   [junit4]   2> 189971 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/timeConfig
   [junit4]   2> 189971 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/timeConfig/managed-schema
   [junit4]   2> 189971 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 189972 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 189972 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 189973 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 189973 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588559659747246080
   [junit4]   2> 189973 INFO  (searcherExecutor-799-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@67d16598[timeConfig_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 189974 INFO  (searcherExecutor-799-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener done.
   [junit4]   2> 189974 INFO  (searcherExecutor-799-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 189977 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 189980 INFO  (searcherExecutor-799-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrCore [timeConfig_shard1_replica_n1] Registered new searcher Searcher@67d16598[timeConfig_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 189980 INFO  (Thread-286) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@319b9577[timeConfig_shard1_replica_n1] main]
   [junit4]   2> 189980 INFO  (searcherExecutor-799-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@319b9577[timeConfig_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 189980 INFO  (searcherExecutor-799-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener done.
   [junit4]   2> 189983 INFO  (searcherExecutor-799-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrCore [timeConfig_shard1_replica_n1] Registered new searcher Searcher@319b9577[timeConfig_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 189986 INFO  (qtp648998840-1405) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler I already have the expected version 0 of config
   [junit4]   2> 189986 INFO  (qtp648998840-1405) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler isStale false , resourceloader org.apache.solr.cloud.ZkSolrResourceLoader
   [junit4]   2> 189986 INFO  (qtp648998840-1405) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.S.Request [timeConfig_shard1_replica_n1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 189989 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Took 216.0ms to set the property overlay to be of version 0 for collection timeConfig
   [junit4]   2> 189990 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrCore [timeConfig_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@70e99b86
   [junit4]   2> 189990 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.timeConfig.shard1.replica_n1, tag=1894357894
   [junit4]   2> 189990 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@253d37ad: rootName = solr_36085, domain = solr.core.timeConfig.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.timeConfig.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@710ec4d9
   [junit4]   2> 189997 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.timeConfig.shard1.leader, tag=1894357894
   [junit4]   2> 189999 INFO  (qtp648998840-1401) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.ZkController Persisted config data to node /configs/timeConfig/params.json 
   [junit4]   2> 189999 INFO  (qtp648998840-1401) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Waiting up to 30 secs for 1 replicas to set the property params to be of version 1 for collection timeConfig
   [junit4]   2> 190000 INFO  (Thread-287) [n:127.0.0.1:36085_solr    ] o.a.s.c.SolrCore config update listener called for core timeConfig_shard1_replica_n1
   [junit4]   2> 190000 INFO  (Thread-287) [n:127.0.0.1:36085_solr    ] o.a.s.c.RequestParams conf resource params.json loaded . version : 1 
   [junit4]   2> 190000 INFO  (Thread-287) [n:127.0.0.1:36085_solr    ] o.a.s.c.RequestParams request params refreshed to version 1
   [junit4]   2> 190001 INFO  (solrHandlerExecutor-803-thread-1-processing-n:127.0.0.1:36085_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 190105 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler I already have the expected version 1 of params
   [junit4]   2> 190105 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler isStale false , resourceloader org.apache.solr.cloud.ZkSolrResourceLoader
   [junit4]   2> 190105 INFO  (qtp648998840-1400) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.S.Request [timeConfig_shard1_replica_n1]  webapp=/solr path=/config/znodeVersion params={params=1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 190107 INFO  (qtp648998840-1401) [n:127.0.0.1:36085_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Took 107.0ms to set the property params to be of version 1 for collection timeConfig
   [junit4]   2> 190108 INFO  (qtp2145550292-1408) [n:127.0.0.1:39429_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=timeConfig&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 190113 INFO  (OverseerThreadFactory-788-thread-2-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 190113 INFO  (OverseerCollectionConfigSetProcessor-99284978467930118-127.0.0.1:36085_solr-n_0000000000) [n:127.0.0.1:36085_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 190114 INFO  (qtp648998840-1402) [n:127.0.0.1:36085_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.timeConfig.shard1.replica_n1, tag=null
   [junit4]   2> 190114 INFO  (qtp648998840-1402) [n:127.0.0.1:36085_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@24247ec5: rootName = solr_36085, domain = solr.core.timeConfig.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.timeConfig.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@710ec4d9
   [junit4]   2> 190122 INFO  (qtp648998840-1402) [n:127.0.0.1:36085_solr    ] o.a.s.c.SolrCore [timeConfig_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4b849afe
   [junit4]   2> 190122 INFO  (qtp648998840-1402) [n:127.0.0.1:36085_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.timeConfig.shard1.replica_n1, tag=1266981630
   [junit4]   2> 190122 INFO  (qtp648998840-1402) [n:127.0.0.1:36085_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.timeConfig.shard1.leader, tag=1266981630
   [junit4]   2> 190124 INFO  (qtp648998840-1402) [n:127.0.0.1:36085_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=timeConfig_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=10
   [junit4]   2> 190829 INFO  (qtp2145550292-1408) [n:127.0.0.1:39429_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=timeConfig&action=DELETE&wt=javabin&version=2} status=0 QTime=721
   [junit4]   2> 190831 INFO  (qtp2145550292-1410) [n:127.0.0.1:39429_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&property.timePartitionAliasName=myalias&collection.configName=timeConfig&name=myalias_2017-10-23&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 190833 INFO  (OverseerThreadFactory-788-thread-3-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.CreateCollectionCmd Create collection myalias_2017-10-23
   [junit4]   2> 190833 INFO  (OverseerCollectionConfigSetProcessor-99284978467930118-127.0.0.1:36085_solr-n_0000000000) [n:127.0.0.1:36085_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 191042 INFO  (OverseerStateUpdate-99284978467930118-127.0.0.1:36085_solr-n_0000000000) [n:127.0.0.1:36085_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"myalias_2017-10-23",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"myalias_2017-10-23_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:36085/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 191244 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=timeConfig&newCollection=true&collection=myalias_2017-10-23&version=2&replicaType=NRT&coreNodeName=core_node2&property.timePartitionAliasName=myalias&name=myalias_2017-10-23_shard1_replica_n1&action=CREATE&numShards=1&shard=shard1&wt=javabin
   [junit4]   2> 191349 INFO  (zkCallback-263-thread-2-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/myalias_2017-10-23/state.json] for collection [myalias_2017-10-23] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 191349 INFO  (zkCallback-263-thread-3-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/myalias_2017-10-23/state.json] for collection [myalias_2017-10-23] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 192254 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.RequestParams conf resource params.json loaded . version : 1 
   [junit4]   2> 192254 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.RequestParams request params refreshed to version 1
   [junit4]   2> 192255 WARN  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../contrib/extraction/lib
   [junit4]   2> 192255 WARN  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../dist
   [junit4]   2> 192255 WARN  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../contrib/clustering/lib
   [junit4]   2> 192255 WARN  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../dist
   [junit4]   2> 192255 WARN  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../contrib/langid/lib
   [junit4]   2> 192255 WARN  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../dist
   [junit4]   2> 192255 WARN  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../contrib/velocity/lib
   [junit4]   2> 192256 WARN  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../dist filtered by solr-velocity-\d.*\.jar to classpath: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1/../../../../dist
   [junit4]   2> 192261 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 192292 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.s.IndexSchema [myalias_2017-10-23_shard1_replica_n1] Schema name=default-config
   [junit4]   2> 192407 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 192426 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'myalias_2017-10-23_shard1_replica_n1' using configuration from collection myalias_2017-10-23, trusted=true
   [junit4]   2> 192426 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36085.solr.core.myalias_2017-10-23.shard1.replica_n1' (registry 'solr.core.myalias_2017-10-23.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2063cea0
   [junit4]   2> 192427 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 192427 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SolrCore [[myalias_2017-10-23_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/myalias_2017-10-23_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001/tempDir-001/node2/./myalias_2017-10-23_shard1_replica_n1/data/]
   [junit4]   2> 192430 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
   [junit4]   2> 192493 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 192493 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 192494 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 192494 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 192496 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@24dc1c1e[myalias_2017-10-23_shard1_replica_n1] main]
   [junit4]   2> 192496 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/timeConfig
   [junit4]   2> 192496 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/timeConfig
   [junit4]   2> 192497 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/timeConfig/managed-schema
   [junit4]   2> 192497 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 192497 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 192498 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 192500 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 192501 INFO  (searcherExecutor-804-thread-1-processing-n:127.0.0.1:36085_solr x:myalias_2017-10-23_shard1_replica_n1 s:shard1 c:myalias_2017-10-23 r:core_node2) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@24dc1c1e[myalias_2017-10-23_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 192501 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588559662398046208
   [junit4]   2> 192501 INFO  (searcherExecutor-804-thread-1-processing-n:127.0.0.1:36085_solr x:myalias_2017-10-23_shard1_replica_n1 s:shard1 c:myalias_2017-10-23 r:core_node2) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener done.
   [junit4]   2> 192501 INFO  (searcherExecutor-804-thread-1-processing-n:127.0.0.1:36085_solr x:myalias_2017-10-23_shard1_replica_n1 s:shard1 c:myalias_2017-10-23 r:core_node2) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 192502 INFO  (searcherExecutor-804-thread-1-processing-n:127.0.0.1:36085_solr x:myalias_2017-10-23_shard1_replica_n1 s:shard1 c:myalias_2017-10-23 r:core_node2) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SolrCore [myalias_2017-10-23_shard1_replica_n1] Registered new searcher Searcher@24dc1c1e[myalias_2017-10-23_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 192504 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 192504 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 192504 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:36085/solr/myalias_2017-10-23_shard1_replica_n1/
   [junit4]   2> 192504 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 192504 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:36085/solr/myalias_2017-10-23_shard1_replica_n1/ has no replicas
   [junit4]   2> 192505 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 192506 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:36085/solr/myalias_2017-10-23_shard1_replica_n1/ shard1
   [junit4]   2> 192607 INFO  (zkCallback-263-thread-3-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/myalias_2017-10-23/state.json] for collection [myalias_2017-10-23] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 192607 INFO  (zkCallback-263-thread-2-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/myalias_2017-10-23/state.json] for collection [myalias_2017-10-23] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 192657 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 192666 INFO  (qtp648998840-1414) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=timeConfig&newCollection=true&collection=myalias_2017-10-23&version=2&replicaType=NRT&coreNodeName=core_node2&property.timePartitionAliasName=myalias&name=myalias_2017-10-23_shard1_replica_n1&action=CREATE&numShards=1&shard=shard1&wt=javabin} status=0 QTime=1422
   [junit4]   2> 192668 INFO  (qtp2145550292-1410) [n:127.0.0.1:39429_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 192759 INFO  (zkCallback-263-thread-2-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/myalias_2017-10-23/state.json] for collection [myalias_2017-10-23] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 192759 INFO  (zkCallback-263-thread-3-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/myalias_2017-10-23/state.json] for collection [myalias_2017-10-23] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 192834 INFO  (OverseerCollectionConfigSetProcessor-99284978467930118-127.0.0.1:36085_solr-n_0000000000) [n:127.0.0.1:36085_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-que

[...truncated too long message...]

nit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> 193867 INFO  (jetty-closer-245-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@d979554{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 193867 INFO  (jetty-closer-245-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=632361116
   [junit4]   2> 193868 INFO  (jetty-closer-245-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 193868 INFO  (jetty-closer-245-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@110ca537: rootName = solr_39429, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@54bd7dc5
   [junit4]   2> 193868 INFO  (jetty-closer-245-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@1e6a125d{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 193868 INFO  (jetty-closer-245-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1287080737
   [junit4]   2> 193868 INFO  (jetty-closer-245-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 193868 INFO  (jetty-closer-245-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7d883fa5: rootName = solr_36085, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@3a8fbf38
   [junit4]   2> 193884 INFO  (jetty-closer-245-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 193884 INFO  (jetty-closer-245-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@645414f7: rootName = solr_39429, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@5533e3ac
   [junit4]   2> 193885 INFO  (jetty-closer-245-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 193886 INFO  (jetty-closer-245-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2d0261b7: rootName = solr_36085, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@5533e3ac
   [junit4]   2> 193888 INFO  (jetty-closer-245-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 193888 INFO  (jetty-closer-245-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7f6e0a63: rootName = solr_36085, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@64f22567
   [junit4]   2> 193888 INFO  (jetty-closer-245-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 193888 INFO  (jetty-closer-245-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@45f6b559: rootName = solr_39429, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@64f22567
   [junit4]   2> 193889 INFO  (jetty-closer-245-thread-1) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:36085_solr
   [junit4]   2> 193889 INFO  (jetty-closer-245-thread-2) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:39429_solr
   [junit4]   2> 193889 INFO  (jetty-closer-245-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 193890 INFO  (zkCallback-260-thread-1-processing-n:127.0.0.1:39429_solr) [n:127.0.0.1:39429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 193890 INFO  (coreCloseExecutor-809-thread-1) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.c.SolrCore [myalias_2017-10-23_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3c823f76
   [junit4]   2> 193890 INFO  (coreCloseExecutor-809-thread-1) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.myalias_2017-10-23.shard1.replica_n1, tag=1015168886
   [junit4]   2> 193890 INFO  (jetty-closer-245-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 193890 INFO  (zkCallback-263-thread-3-processing-n:127.0.0.1:36085_solr) [n:127.0.0.1:36085_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 193890 INFO  (jetty-closer-245-thread-2) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 193891 INFO  (zkCallback-260-thread-2-processing-n:127.0.0.1:39429_solr) [n:127.0.0.1:39429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 193890 INFO  (coreCloseExecutor-809-thread-1) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5d0466c9: rootName = solr_36085, domain = solr.core.myalias_2017-10-23.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.myalias_2017-10-23.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@432a7fe8
   [junit4]   2> 193892 INFO  (jetty-closer-245-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@64a6309e{/solr,null,UNAVAILABLE}
   [junit4]   2> 193899 INFO  (coreCloseExecutor-809-thread-1) [n:127.0.0.1:36085_solr c:myalias_2017-10-23 s:shard1 r:core_node2 x:myalias_2017-10-23_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.myalias_2017-10-23.shard1.leader, tag=1015168886
   [junit4]   2> 193901 INFO  (jetty-closer-245-thread-1) [    ] o.a.s.c.Overseer Overseer (id=99284978467930118-127.0.0.1:36085_solr-n_0000000000) closing
   [junit4]   2> 193901 INFO  (OverseerStateUpdate-99284978467930118-127.0.0.1:36085_solr-n_0000000000) [n:127.0.0.1:36085_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:36085_solr
   [junit4]   2> 193907 WARN  (OverseerAutoScalingTriggerThread-99284978467930118-127.0.0.1:36085_solr-n_0000000000) [n:127.0.0.1:36085_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 193952 INFO  (jetty-closer-245-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6d273b7{/solr,null,UNAVAILABLE}
   [junit4]   2> 193953 ERROR (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E7B212BFFE4FF2E8]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 193953 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[E7B212BFFE4FF2E8]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42695 42695
   [junit4]   2> 199073 INFO  (Thread-267) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42695 42695
   [junit4]   2> 199077 WARN  (Thread-267) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/aliases.json
   [junit4]   2> 	4	/solr/configs/timeConfig
   [junit4]   2> 	3	/solr/configs/timeConfig/managed-schema
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/collections/timeConfig/state.json
   [junit4]   2> 	5	/solr/collections/myalias_2017-10-23/state.json
   [junit4]   2> 	4	/solr/clusterstate.json
   [junit4]   2> 	4	/solr/clusterprops.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/live_nodes
   [junit4]   2> 	4	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_E7B212BFFE4FF2E8-001
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=false): {}, locale=en-PN, timezone=Africa/Lubumbashi
   [junit4]   2> NOTE: Linux 4.10.0-40-generic amd64/Oracle Corporation 9.0.1 (64-bit)/cpus=8,threads=1,free=279581832,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestTlogReplica, TestNumericTerms32, BlockJoinFacetDistribTest, ReplicaListTransformerTest, AnalyticsQueryTest, AtomicUpdatesTest, TestOverriddenPrefixQueryForCustomFieldType, TestBinaryResponseWriter, TestFieldCacheVsDocValues, SolrPluginUtilsTest, BasicFunctionalityTest, StatsReloadRaceTest, SpatialFilterTest, TestSerializedLuceneMatchVersion, BlockDirectoryTest, RemoteQueryErrorTest, TestDFRSimilarityFactory, TestSort, HdfsTlogReplayBufferedWhileIndexingTest, TestFieldCacheReopen, TimeRoutedAliasUpdateProcessorTest]
   [junit4] Completed [47/764 (1!)] on J0 in 13.23s, 2 tests, 1 failure <<< FAILURES!

[...truncated 43277 lines...]

[JENKINS] Lucene-Solr-7.x-Linux (32bit/jdk1.8.0_144) - Build # 1107 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1107/
Java: 32bit/jdk1.8.0_144 -client -XX:+UseSerialGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.schema.ManagedSchemaRoundRobinCloudTest

Error Message:
Error from server at https://127.0.0.1:41313/solr: create the collection time out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:41313/solr: create the collection time out:180s
	at __randomizedtesting.SeedInfo.seed([627E21C664583B7F]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.schema.ManagedSchemaRoundRobinCloudTest.setupCluster(ManagedSchemaRoundRobinCloudTest.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:41615/ofs

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:41615/ofs
	at __randomizedtesting.SeedInfo.seed([627E21C664583B7F:EA2A1E1CCAA45687]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:320)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 43 more


FAILED:  org.apache.solr.cloud.TestDistribDocBasedVersion.test

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:46295

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:46295
	at __randomizedtesting.SeedInfo.seed([627E21C664583B7F:EA2A1E1CCAA45687]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:320)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 43 more




Build Log:
[...truncated 12535 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestDistribDocBasedVersion
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestDistribDocBasedVersion_627E21C664583B7F-001/init-core-data-001
   [junit4]   2> 382065 WARN  (SUITE-TestDistribDocBasedVersion-seed#[627E21C664583B7F]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=33 numCloses=33
   [junit4]   2> 382065 INFO  (SUITE-TestDistribDocBasedVersion-seed#[627E21C664583B7F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 382066 INFO  (SUITE-TestDistribDocBasedVersion-seed#[627E21C664583B7F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 382067 INFO  (SUITE-TestDistribDocBasedVersion-seed#[627E21C664583B7F]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 382067 INFO  (SUITE-TestDistribDocBasedVersion-seed#[627E21C664583B7F]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 382069 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 382069 INFO  (Thread-346) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 382069 INFO  (Thread-346) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 382071 ERROR (Thread-346) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 382169 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.ZkTestServer start zk server on port:41889
   [junit4]   2> 382171 INFO  (zkConnectionManagerCallback-571-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 382174 INFO  (zkConnectionManagerCallback-573-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 382178 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-externalversionconstraint.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 382179 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 382180 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 382181 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 382182 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 382182 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 382183 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 382183 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 382184 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 382185 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 382185 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 382190 INFO  (zkConnectionManagerCallback-576-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 382192 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 382293 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 382294 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@28437f{/,null,AVAILABLE}
   [junit4]   2> 382296 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@c1ab9b{SSL,[ssl, http/1.1]}{127.0.0.1:46295}
   [junit4]   2> 382296 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.e.j.s.Server Started @383533ms
   [junit4]   2> 382296 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestDistribDocBasedVersion_627E21C664583B7F-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=46295, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestDistribDocBasedVersion_627E21C664583B7F-001/control-001/cores}
   [junit4]   2> 382296 ERROR (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 382296 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 382296 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 382297 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 382297 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-03T05:18:40.414Z
   [junit4]   2> 382298 INFO  (zkConnectionManagerCallback-578-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 382299 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 382299 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestDistribDocBasedVersion_627E21C664583B7F-001/control-001/solr.xml
   [junit4]   2> 382308 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 382308 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 382320 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1787ba2, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 382324 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41889/solr
   [junit4]   2> 382325 INFO  (zkConnectionManagerCallback-582-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 382327 INFO  (zkConnectionManagerCallback-584-thread-1-processing-n:127.0.0.1:46295_) [n:127.0.0.1:46295_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 382368 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [n:127.0.0.1:46295_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 382369 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [n:127.0.0.1:46295_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46295_
   [junit4]   2> 382369 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [n:127.0.0.1:46295_    ] o.a.s.c.Overseer Overseer (id=99284203614175237-127.0.0.1:46295_-n_0000000000) starting
   [junit4]   2> 382373 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [n:127.0.0.1:46295_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46295_
   [junit4]   2> 382374 INFO  (OverseerStateUpdate-99284203614175237-127.0.0.1:46295_-n_0000000000) [n:127.0.0.1:46295_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 382517 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [n:127.0.0.1:46295_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1787ba2
   [junit4]   2> 382528 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [n:127.0.0.1:46295_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1787ba2
   [junit4]   2> 382528 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [n:127.0.0.1:46295_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1787ba2
   [junit4]   2> 382530 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [n:127.0.0.1:46295_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestDistribDocBasedVersion_627E21C664583B7F-001/control-001/cores
   [junit4]   2> 382560 INFO  (zkConnectionManagerCallback-590-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 382561 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 382561 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41889/solr ready
   [junit4]   2> 382610 INFO  (qtp9343557-2282) [n:127.0.0.1:46295_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:46295_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 382611 INFO  (OverseerThreadFactory-1081-thread-1-processing-n:127.0.0.1:46295_) [n:127.0.0.1:46295_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 382719 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 382719 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 382822 INFO  (zkCallback-583-thread-1-processing-n:127.0.0.1:46295_) [n:127.0.0.1:46295_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 383760 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 383791 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 383885 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 383903 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 383904 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1787ba2
   [junit4]   2> 383908 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 383908 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestDistribDocBasedVersion_627E21C664583B7F-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestDistribDocBasedVersion_627E21C664583B7F-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 383911 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=13, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 384026 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 384026 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 384027 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 384027 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 384029 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=75.6220703125, floorSegmentMB=2.1572265625, forceMergeDeletesPctAllowed=9.433735447934225, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 384029 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1b03dc9[control_collection_shard1_replica_n1] main]
   [junit4]   2> 384030 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 384031 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 384031 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 384033 INFO  (searcherExecutor-1084-thread-1-processing-n:127.0.0.1:46295_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@1b03dc9[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 384033 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588547259885158400
   [junit4]   2> 384037 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 384037 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 384037 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:46295/control_collection_shard1_replica_n1/
   [junit4]   2> 384037 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 384037 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:46295/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 384037 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 384038 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:46295/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 442667 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.ChaosMonkey monkey: stop jetty! 46295
   [junit4]   2> 442667 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=15150168
   [junit4]   2> 442667 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 442668 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@17ce72: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@339cbc
   [junit4]   2> 442673 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 442673 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1aa8130: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@6a3268
   [junit4]   2> 442678 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 442678 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@16e03a5: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@bbde7b
   [junit4]   2> 442680 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:46295_
   [junit4]   2> 442682 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 442683 INFO  (zkCallback-583-thread-2-processing-n:127.0.0.1:46295_) [n:127.0.0.1:46295_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 442684 INFO  (coreCloseExecutor-1089-thread-1) [n:127.0.0.1:46295_ c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1b1ce36
   [junit4]   2> 442684 INFO  (coreCloseExecutor-1089-thread-1) [n:127.0.0.1:46295_ c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.control_collection.shard1.replica_n1, tag=28429878
   [junit4]   2> 442684 INFO  (coreCloseExecutor-1089-thread-1) [n:127.0.0.1:46295_ c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@85c0e1: rootName = null, domain = solr.core.control_collection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.control_collection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@8a13b6
   [junit4]   2> 442694 INFO  (coreCloseExecutor-1089-thread-1) [n:127.0.0.1:46295_ c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.control_collection.shard1.leader, tag=28429878
   [junit4]   2> 453393 INFO  (ScheduledTrigger-1079-thread-1) [n:127.0.0.1:46295_    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 453394 INFO  (AutoscalingActionExecutor-1080-thread-1-processing-n:127.0.0.1:46295_) [n:127.0.0.1:46295_    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 2657933483 sessionWrapper.createTime 2657933483665519, this.sessionWrapper.createTime 2657933483665519 
   [junit4]   2> 453394 INFO  (AutoscalingActionExecutor-1080-thread-1-processing-n:127.0.0.1:46295_) [n:127.0.0.1:46295_    ] o.a.s.c.a.ExecutePlanAction No operations to execute for event: {
   [junit4]   2>   "id":"9715e08ce2191T8suuzuu59z7zsmdr0jb3f3475",
   [junit4]   2>   "source":".auto_add_replicas",
   [junit4]   2>   "eventTime":2657923478987153,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2657923478987153],
   [junit4]   2>     "_enqueue_time_":2657933481515656,
   [junit4]   2>     "nodeNames":["127.0.0.1:46295_"]}}
   [junit4]   2> 562612 ERROR (qtp9343557-2282) [n:127.0.0.1:46295_    ] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: create the collection time out:180s
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:314)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:246)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:224)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 562612 INFO  (qtp9343557-2282) [n:127.0.0.1:46295_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:46295_&wt=javabin&version=2} status=500 QTime=180002
   [junit4]   2> 562613 ERROR (qtp9343557-2282) [n:127.0.0.1:46295_    ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: create the collection time out:180s
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:314)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:246)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:224)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 982803 ERROR (OverseerThreadFactory-1081-thread-1-processing-n:127.0.0.1:46295_) [n:127.0.0.1:46295_    ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: https://127.0.0.1:46295
   [junit4]   2> org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:46295
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]   2> 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]   2> 	at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:172)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]   2> 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]   2> 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]   2> 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]   2> 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]   2> 	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
   [junit4]   2> 	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
   [junit4]   2> 	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
   [junit4]   2> 	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
   [junit4]   2> 	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
   [junit4]   2> 	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   [junit4]   2> 	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   [junit4]   2> 	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]   2> 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]   2> 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]   2> 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]   2> 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:118)
   [junit4]   2> 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]   2> 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]   2> 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]   2> 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
   [junit4]   2> 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]   2> 	... 12 more
   [junit4]   2> 982804 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.Overseer Overseer (id=99284203614175237-127.0.0.1:46295_-n_0000000000) closing
   [junit4]   2> 982805 ERROR (OverseerThreadFactory-1081-thread-1-processing-n:127.0.0.1:46295_) [n:127.0.0.1:46295_    ] o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [control_collection].
   [junit4]   2> 982805 INFO  (OverseerStateUpdate-99284203614175237-127.0.0.1:46295_-n_0000000000) [n:127.0.0.1:46295_    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:46295_
   [junit4]   2> 982806 INFO  (OverseerThreadFactory-1081-thread-1-processing-n:127.0.0.1:46295_) [n:127.0.0.1:46295_    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 1012843 INFO  (zkCallback-583-thread-5-processing-n:127.0.0.1:46295_) [n:127.0.0.1:46295_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1012846 ERROR (OverseerThreadFactory-1081-thread-1-processing-n:127.0.0.1:46295_) [n:127.0.0.1:46295_    ] o.a.s.c.OverseerCollectionMessageHandler Collection: control_collection operation: create failed:org.apache.solr.common.SolrException: Could not fully remove collection: control_collection
   [junit4]   2> 	at org.apache.solr.cloud.DeleteCollectionCmd.call(DeleteCollectionCmd.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionMessageHandler.cleanupCollection(OverseerCollectionMessageHandler.java:688)
   [junit4]   2> 	at org.apache.solr.cloud.CreateCollectionCmd.call(CreateCollectionCmd.java:240)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:242)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:464)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 1012859 INFO  (OverseerThreadFactory-1081-thread-1-processing-n:127.0.0.1:46295_) [n:127.0.0.1:46295_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1012860 WARN  (OverseerAutoScalingTriggerThread-99284203614175237-127.0.0.1:46295_-n_0000000000) [n:127.0.0.1:46295_    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1012864 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@c1ab9b{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1012864 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@28437f{/,null,UNAVAILABLE}
   [junit4]   2> 1012888 ERROR (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController Error getting leader from zk
   [junit4]   2> org.apache.solr.common.SolrException: No registered leader was found after waiting for 1220000ms , collection: control_collection slice: shard1 saw state=null with live_nodes=[]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:771)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:736)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1178)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1079)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1014)
   [junit4]   2> 	at org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:181)
   [junit4]   2> 	at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:208)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:888)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1049)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:950)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1012889 ERROR (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkContainer :org.apache.solr.common.SolrException: Error getting leader from zk for shard shard1
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1207)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1079)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1014)
   [junit4]   2> 	at org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:181)
   [junit4]   2> 	at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:208)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:888)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1049)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:950)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: No registered leader was found after waiting for 1220000ms , collection: control_collection slice: shard1 saw state=null with live_nodes=[]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:771)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:736)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1178)
   [junit4]   2> 	... 44 more
   [junit4]   2> 
   [junit4]   2> 1012890 INFO  (qtp9343557-2278) [n:127.0.0.1:46295_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=630171
   [junit4]   2> 1012890 ERROR (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1012890 INFO  (TEST-TestDistribDocBasedVersion.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41889 41889
   [junit4]   2> 1018006 INFO  (Thread-346) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41889 41889
   [junit4]   2> 1018007 WARN  (Thread-346) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/aliases.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/clusterprops.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/live_nodes
   [junit4]   2> 	2	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDistribDocBasedVersion -Dtests.method=test -Dtests.seed=627E21C664583B7F -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=bg -Dtests.timezone=Africa/Lagos -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR    636s J2 | TestDistribDocBasedVersion.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:46295
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([627E21C664583B7F:EA2A1E1CCAA45687]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:320)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]    > 	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
   [junit4]    > 	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
   [junit4]    > 	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
   [junit4]    > 	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
   [junit4]    > 	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]    > 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]    > 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]    > 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]    > 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]    > 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]    > 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
   [junit4]    > 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]    > 	... 43 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestDistribDocBasedVersion_627E21C664583B7F-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=true): {}, locale=bg, timezone=Africa/Lagos
   [junit4]   2> NOTE: Linux 4.10.0-40-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=1,free=54820864,total=195878912
   [junit4]   2> NOTE: All tests run in this JVM: [RuleEngineTest, UUIDUpdateProcessorFallbackTest, TestCloudManagedSchema, TestLFUCache, CdcrUpdateLogTest, WordBreakSolrSpellCheckerTest, TestLMJelinekMercerSimilarityFactory, TestExportWriter, DocExpirationUpdateProcessorFactoryTest, TestTrieFacet, TestExtendedDismaxParser, ZkNodePropsTest, TestRealTimeGet, HighlighterMaxOffsetTest, TestSegmentSorting, DirectSolrConnectionTest, IndexSchemaTest, ReplicationFactorTest, LeaderElectionTest, TestTrie, EchoParamsTest, AtomicUpdateProcessorFactoryTest, CSVRequestHandlerTest, TestPolicyCloud, CollectionReloadTest, TestIndexSearcher, TestPullReplica, IndexBasedSpellCheckerTest, TestLMDirichletSimilarityFactory, TestSolrXml, TestQueryTypes, TestNumericRangeQuery64, TestNamedUpdateProcessors, TestDistribDocBasedVersion]
   [junit4] Completed [254/764 (1!)] on J2 in 635.97s, 1 test, 1 error <<< FAILURES!

[...truncated 933 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_627E21C664583B7F-001/init-core-data-001
   [junit4]   2> 1761265 WARN  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[627E21C664583B7F]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 1761265 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[627E21C664583B7F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1761266 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[627E21C664583B7F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1761266 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[627E21C664583B7F]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1761266 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[627E21C664583B7F]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /ofs/
   [junit4]   2> 1761266 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.ChaosMonkeySafeLeaderWithPullReplicasTest Starting ChaosMonkey test with 1 shards and 3 nodes
   [junit4]   2> 1761270 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1761270 INFO  (Thread-5814) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1761270 INFO  (Thread-5814) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1761272 ERROR (Thread-5814) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1761370 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.ZkTestServer start zk server on port:44305
   [junit4]   2> 1761376 INFO  (zkConnectionManagerCallback-3044-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1761384 INFO  (zkConnectionManagerCallback-3046-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1761394 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1761396 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1761398 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1761400 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1761401 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1761403 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1761405 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1761407 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1761409 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1761411 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1761411 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1761411 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 1761483 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1761483 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@9bff00{/ofs,null,AVAILABLE}
   [junit4]   2> 1761485 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@17ba23e{HTTP/1.1,[http/1.1]}{127.0.0.1:?????}
   [junit4]   2> 1761485 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.e.j.s.Server Started @???????ms
   [junit4]   2> 1761485 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_627E21C664583B7F-001/tempDir-001/control/data, hostContext=/ofs, hostPort=41615, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_627E21C664583B7F-001/control-001/cores}
   [junit4]   2> 1761485 ERROR (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1761485 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 1761485 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1761485 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1761485 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-03T05:41:39.555Z
   [junit4]   2> 1761487 INFO  (zkConnectionManagerCallback-3048-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1761488 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1761488 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_627E21C664583B7F-001/control-001/solr.xml
   [junit4]   2> 1761494 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1761494 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1761496 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5e0c15, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1761498 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44305/solr
   [junit4]   2> 1761499 INFO  (zkConnectionManagerCallback-3052-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1761501 INFO  (zkConnectionManagerCallback-3054-thread-1-processing-n:127.0.0.1:41615_ofs) [n:127.0.0.1:41615_ofs    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1761533 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [n:127.0.0.1:41615_ofs    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1761533 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [n:127.0.0.1:41615_ofs    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41615_ofs
   [junit4]   2> 1761534 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [n:127.0.0.1:41615_ofs    ] o.a.s.c.Overseer Overseer (id=99284293998411780-127.0.0.1:41615_ofs-n_0000000000) starting
   [junit4]   2> 1761538 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [n:127.0.0.1:41615_ofs    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41615_ofs
   [junit4]   2> 1761542 INFO  (zkCallback-3053-thread-1-processing-n:127.0.0.1:41615_ofs) [n:127.0.0.1:41615_ofs    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1761670 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [n:127.0.0.1:41615_ofs    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e0c15
   [junit4]   2> 1761692 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [n:127.0.0.1:41615_ofs    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e0c15
   [junit4]   2> 1761693 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [n:127.0.0.1:41615_ofs    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e0c15
   [junit4]   2> 1761694 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[627E21C664583B7F]) [n:127.0.0.1:41615_ofs    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lu

[...truncated too long message...]

	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1178)
   [junit4]   2> 	... 44 more
   [junit4]   2> 
   [junit4]   2> 3231824 INFO  (qtp2742096-22944) [n:127.0.0.1:41313_solr c:managed_coll s:shard1 r:core_node3 x:managed_coll_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=cloud-managed&newCollection=true&name=managed_coll_shard1_replica_n1&action=CREATE&numShards=2&collection=managed_coll&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=209747
   [junit4]   2> 3231825 ERROR (SUITE-ManagedSchemaRoundRobinCloudTest-seed#[627E21C664583B7F]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3231827 INFO  (SUITE-ManagedSchemaRoundRobinCloudTest-seed#[627E21C664583B7F]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41267 41267
   [junit4]   2> 3237238 INFO  (Thread-7215) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41267 41267
   [junit4]   2> 3237239 WARN  (Thread-7215) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/aliases.json
   [junit4]   2> 	2	/solr/configs/cloud-managed
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 	2	/solr/configs/cloud-managed/managed-schema
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/collections/managed_coll/state.json
   [junit4]   2> 	3	/solr/clusterstate.json
   [junit4]   2> 	3	/solr/clusterprops.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 	3	/solr/collections
   [junit4]   2> 
   [junit4]   2> 3237240 INFO  (SUITE-ManagedSchemaRoundRobinCloudTest-seed#[627E21C664583B7F]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=414, maxMBSortInHeap=7.976452469000723, sim=RandomSimilarity(queryNorm=true): {}, locale=en-ZA, timezone=Asia/Damascus
   [junit4]   2> NOTE: Linux 4.10.0-40-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=1,free=90414032,total=409894912
   [junit4]   2> NOTE: All tests run in this JVM: [TestApiFramework, ChangedSchemaMergeTest, TestSolrConfigHandler, SolrCLIZkUtilsTest, PrimUtilsTest, TestRandomCollapseQParserPlugin, ConvertedLegacyTest, TestFieldTypeResource, TestSolrQueryResponse, BufferStoreTest, HdfsUnloadDistributedZkTest, ParsingFieldUpdateProcessorsTest, PeerSyncWithIndexFingerprintCachingTest, SchemaVersionSpecificBehaviorTest, TestRawResponseWriter, BasicAuthStandaloneTest, TestExecutePlanAction, PingRequestHandlerTest, TestCursorMarkWithoutUniqueKey, DocValuesMultiTest, CdcrReplicationHandlerTest, TestSimpleTrackingShardHandler, AddReplicaTest, DocumentAnalysisRequestHandlerTest, TestPKIAuthenticationPlugin, TestLazyCores, TestJsonFacets, TestSlowCompositeReaderWrapper, HdfsAutoAddReplicasIntegrationTest, SolrJmxReporterCloudTest, OverriddenZkACLAndCredentialsProvidersTest, SparseHLLTest, TlogReplayBufferedWhileIndexingTest, TestSchemaManager, TestRTGBase, SolrCloudReportersTest, TestReplicaProperties, DistributedQueryComponentCustomSortTest, OpenExchangeRatesOrgProviderTest, DistributedFacetPivotSmallAdvancedTest, TestBulkSchemaConcurrent, HdfsBasicDistributedZk2Test, TestFieldCacheWithThreads, TestBackupRepositoryFactory, TestCloudNestedDocsSort, ImplicitSnitchTest, SolrRequestParserTest, TestXmlQParserPlugin, TestOmitPositions, TestDelegationWithHadoopAuth, TestDocSet, TestHttpShardHandlerFactory, SimpleMLTQParserTest, SuggesterFSTTest, CloudExitableDirectoryReaderTest, TestExceedMaxTermLength, TestSubQueryTransformerCrossCore, ReplaceNodeTest, TestHdfsBackupRestoreCore, TestCollationFieldDocValues, LukeRequestHandlerTest, TestDeleteCollectionOnDownNodes, TemplateUpdateProcessorTest, TestFieldCollectionResource, SignatureUpdateProcessorFactoryTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestSolr4Spatial, TestFastLRUCache, CircularListTest, LegacyCloudClusterPropTest, CursorMarkTest, TestSizeLimitedDistributedMap, NotRequiredUniqueKeyTest, ClassificationUpdateProcessorFactoryTest, TestLuceneMatchVersion, PolyFieldTest, TestJmxIntegration, TestLeaderElectionZkExpiry, TestCollapseQParserPlugin, ZkSolrClientTest, TestDocumentBuilder, TestScoreJoinQPScore, TestBlendedInfixSuggestions, ShardRoutingTest, ConcurrentDeleteAndCreateCollectionTest, PeerSyncReplicationTest, TestWriterPerf, TestEmbeddedSolrServerAdminHandler, TestConfigSetProperties, TestPrepRecovery, TestFaceting, SpellingQueryConverterTest, FacetPivotSmallTest, BigEndianAscendingWordSerializerTest, HttpPartitionTest, DistribDocExpirationUpdateProcessorTest, TestRecoveryHdfs, TestStressInPlaceUpdates, DebugComponentTest, TestGraphMLResponseWriter, TestCustomStream, TestCollectionAPI, TestNumericRangeQuery32, TestManagedSynonymGraphFilterFactory, CdcrBidirectionalTest, TestReloadAndDeleteDocs, LeaderFailureAfterFreshStartTest, SubstringBytesRefFilterTest, SecurityConfHandlerTest, MoveReplicaHDFSFailoverTest, MoreLikeThisHandlerTest, TestDefaultStatsCache, NumberUtilsTest, BlockJoinFacetRandomTest, ZkCLITest, TestNodeAddedTrigger, PreAnalyzedFieldTest, JavabinLoaderTest, CloneFieldUpdateProcessorFactoryTest, DOMUtilTest, SolrIndexConfigTest, TestLegacyNumericUtils, ExitableDirectoryReaderTest, ChaosMonkeySafeLeaderTest, SyncSliceTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, TestRandomFaceting, ShardRoutingCustomTest, TestDistributedGrouping, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, TestStressReorder, TestStressVersions, TestRangeQuery, PeerSyncTest, BasicFunctionalityTest, DirectUpdateHandlerTest, HighlighterTest, SoftAutoCommitTest, TestSystemIdResolver, ClusterStateTest, TestSolrJ, TestLRUCache, TestUtils, SliceStateTest, DistributedMLTComponentTest, CursorPagingTest, DistributedIntervalFacetingTest, TestHighlightDedupGrouping, TestTolerantSearch, TestLuceneIndexBackCompat, TestEmbeddedSolrServerConstructors, TestEmbeddedSolrServerSchemaAPI, TestJettySolrRunner, AliasIntegrationTest, AssignTest, AsyncCallRequestStatusResponseTest, ChaosMonkeySafeLeaderWithPullReplicasTest, TestLocalFSCloudBackupRestore, TestRandomFlRTGCloud, TestRequestStatusCollectionAPI, TestSolrCloudWithKerberosAlt, TestTolerantUpdateProcessorCloud, ZkFailoverTest, NodeAddedTriggerTest, SystemLogListenerTest, TestComputePlanAction, TestNodeLostTrigger, TestSimDistributedQueue, BaseCdcrDistributedZkTest, CdcrBootstrapTest, HdfsSyncSliceTest, HdfsThreadLeakTest, HdfsTlogReplayBufferedWhileIndexingTest, StressHdfsTest, BlobRepositoryCloudTest, CachingDirectoryFactoryTest, DirectoryFactoryTest, TestDynamicLoading, TestSolrConfigHandlerConcurrent, TestSystemCollAutoCreate, DistributedQueryComponentOptimizationTest, SuggestComponentTest, TestLegacyFieldReuse, TestLegacyTerms, TestMultiValuedNumericRangeQuery, SolrCoreMetricManagerTest, SolrGangliaReporterTest, SolrShardReporterTest, TestStreamBody, TestMacroExpander, TestMacros, JSONWriterTest, TestCustomDocTransformer, TestJavabinTupleStreamParser, TestRetrieveFieldsOptimizer, TestChildDocTransformer, TestSubQueryTransformer, TestManagedResourceStorage, TestCopyFieldCollectionResource, TestDynamicFieldCollectionResource, TestDynamicFieldResource, TestFieldTypeCollectionResource, TestSchemaSimilarityResource, TestSchemaVersionResource, TestSerializedLuceneMatchVersion, TestUniqueKeyFieldResource, TestManagedStopFilterFactory, BadCopyFieldTest, BooleanFieldTest, CurrencyFieldTypeTest, DocValuesMissingTest, ManagedSchemaRoundRobinCloudTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ManagedSchemaRoundRobinCloudTest -Dtests.seed=627E21C664583B7F -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-ZA -Dtests.timezone=Asia/Damascus -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | ManagedSchemaRoundRobinCloudTest (suite) <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:41313/solr: create the collection time out:180s
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([627E21C664583B7F]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
   [junit4]    > 	at org.apache.solr.schema.ManagedSchemaRoundRobinCloudTest.setupCluster(ManagedSchemaRoundRobinCloudTest.java:50)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4] Completed [748/764 (3!)] on J1 in 216.20s, 0 tests, 1 error <<< FAILURES!

[...truncated 43924 lines...]

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-9.0.1) - Build # 1106 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1106/
Java: 64bit/jdk-9.0.1 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue

Error Message:
action wasn't interrupted

Stack Trace:
java.lang.AssertionError: action wasn't interrupted
	at __randomizedtesting.SeedInfo.seed([5BFC67CD2302993B:924925632A655FCE]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue(TriggerIntegrationTest.java:711)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.metrics.reporters.solr.SolrShardReporterTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:44667

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:44667
	at __randomizedtesting.SeedInfo.seed([5BFC67CD2302993B:D3A858178DFEF4C3]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:320)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.base/java.net.SocketInputStream.socketRead0(Native Method)
	at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 42 more




Build Log:
[...truncated 12639 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
   [junit4]   2> 1023148 INFO  (SUITE-TriggerIntegrationTest-seed#[5BFC67CD2302993B]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5BFC67CD2302993B-001/init-core-data-001
   [junit4]   2> 1023148 WARN  (SUITE-TriggerIntegrationTest-seed#[5BFC67CD2302993B]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=6 numCloses=6
   [junit4]   2> 1023148 INFO  (SUITE-TriggerIntegrationTest-seed#[5BFC67CD2302993B]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1023149 INFO  (SUITE-TriggerIntegrationTest-seed#[5BFC67CD2302993B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
   [junit4]   2> 1023149 INFO  (SUITE-TriggerIntegrationTest-seed#[5BFC67CD2302993B]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5BFC67CD2302993B-001/tempDir-001
   [junit4]   2> 1023149 INFO  (SUITE-TriggerIntegrationTest-seed#[5BFC67CD2302993B]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1023158 INFO  (Thread-1536) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1023158 INFO  (Thread-1536) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1023159 ERROR (Thread-1536) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1023249 INFO  (SUITE-TriggerIntegrationTest-seed#[5BFC67CD2302993B]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:41913
   [junit4]   2> 1023251 INFO  (zkConnectionManagerCallback-1526-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1023254 INFO  (jetty-launcher-1523-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1023254 INFO  (jetty-launcher-1523-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1023255 INFO  (jetty-launcher-1523-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@53684d6a{/solr,null,AVAILABLE}
   [junit4]   2> 1023255 INFO  (jetty-launcher-1523-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@43fd59d5{/solr,null,AVAILABLE}
   [junit4]   2> 1023260 INFO  (jetty-launcher-1523-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6c7fa7d9{SSL,[ssl, http/1.1]}{127.0.0.1:33721}
   [junit4]   2> 1023260 INFO  (jetty-launcher-1523-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@cb2b402{SSL,[ssl, http/1.1]}{127.0.0.1:41887}
   [junit4]   2> 1023260 INFO  (jetty-launcher-1523-thread-2) [    ] o.e.j.s.Server Started @1025274ms
   [junit4]   2> 1023260 INFO  (jetty-launcher-1523-thread-1) [    ] o.e.j.s.Server Started @1025274ms
   [junit4]   2> 1023260 INFO  (jetty-launcher-1523-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33721}
   [junit4]   2> 1023260 INFO  (jetty-launcher-1523-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41887}
   [junit4]   2> 1023261 ERROR (jetty-launcher-1523-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1023261 ERROR (jetty-launcher-1523-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1023261 INFO  (jetty-launcher-1523-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 1023261 INFO  (jetty-launcher-1523-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 1023261 INFO  (jetty-launcher-1523-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1023261 INFO  (jetty-launcher-1523-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1023261 INFO  (jetty-launcher-1523-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1023261 INFO  (jetty-launcher-1523-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1023261 INFO  (jetty-launcher-1523-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-03T01:49:57.063196Z
   [junit4]   2> 1023261 INFO  (jetty-launcher-1523-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-03T01:49:57.063233Z
   [junit4]   2> 1023262 INFO  (zkConnectionManagerCallback-1530-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1023262 INFO  (jetty-launcher-1523-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1023263 INFO  (zkConnectionManagerCallback-1528-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1023264 INFO  (jetty-launcher-1523-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1023281 INFO  (jetty-launcher-1523-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41913/solr
   [junit4]   2> 1023282 INFO  (zkConnectionManagerCallback-1534-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1023283 INFO  (jetty-launcher-1523-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41913/solr
   [junit4]   2> 1023283 INFO  (zkConnectionManagerCallback-1538-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1023283 INFO  (zkConnectionManagerCallback-1540-thread-1-processing-n:127.0.0.1:33721_solr) [n:127.0.0.1:33721_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1023285 INFO  (zkConnectionManagerCallback-1542-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1023314 INFO  (jetty-launcher-1523-thread-2) [n:127.0.0.1:33721_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1023314 INFO  (jetty-launcher-1523-thread-2) [n:127.0.0.1:33721_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33721_solr
   [junit4]   2> 1023314 INFO  (jetty-launcher-1523-thread-1) [n:127.0.0.1:41887_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1023315 INFO  (jetty-launcher-1523-thread-2) [n:127.0.0.1:33721_solr    ] o.a.s.c.Overseer Overseer (id=99283382892036101-127.0.0.1:33721_solr-n_0000000000) starting
   [junit4]   2> 1023315 INFO  (jetty-launcher-1523-thread-1) [n:127.0.0.1:41887_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41887_solr
   [junit4]   2> 1023316 INFO  (zkCallback-1541-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1023326 INFO  (zkCallback-1539-thread-1-processing-n:127.0.0.1:33721_solr) [n:127.0.0.1:33721_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1023342 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1023342 INFO  (jetty-launcher-1523-thread-2) [n:127.0.0.1:33721_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33721_solr
   [junit4]   2> 1023342 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 1023343 INFO  (zkCallback-1541-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1023343 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:41887_solr]
   [junit4]   2> 1023343 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 1023343 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 1023343 INFO  (zkCallback-1539-thread-1-processing-n:127.0.0.1:33721_solr) [n:127.0.0.1:33721_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1023344 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1023344 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 1023344 DEBUG (ScheduledTrigger-3378-thread-1) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1023360 INFO  (jetty-launcher-1523-thread-1) [n:127.0.0.1:41887_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41887.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1023366 INFO  (jetty-launcher-1523-thread-2) [n:127.0.0.1:33721_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33721.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1023368 INFO  (jetty-launcher-1523-thread-1) [n:127.0.0.1:41887_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41887.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1023368 INFO  (jetty-launcher-1523-thread-1) [n:127.0.0.1:41887_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41887.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1023369 INFO  (jetty-launcher-1523-thread-1) [n:127.0.0.1:41887_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5BFC67CD2302993B-001/tempDir-001/node1/.
   [junit4]   2> 1023371 INFO  (jetty-launcher-1523-thread-2) [n:127.0.0.1:33721_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33721.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1023371 INFO  (jetty-launcher-1523-thread-2) [n:127.0.0.1:33721_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33721.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1023371 INFO  (jetty-launcher-1523-thread-2) [n:127.0.0.1:33721_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5BFC67CD2302993B-001/tempDir-001/node2/.
   [junit4]   2> 1023406 INFO  (zkConnectionManagerCallback-1548-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1023408 INFO  (zkConnectionManagerCallback-1552-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1023408 INFO  (SUITE-TriggerIntegrationTest-seed#[5BFC67CD2302993B]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1023409 INFO  (SUITE-TriggerIntegrationTest-seed#[5BFC67CD2302993B]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41913/solr ready
   [junit4]   2> 1023418 INFO  (TEST-TriggerIntegrationTest.testSetProperties-seed#[5BFC67CD2302993B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSetProperties
   [junit4]   2> 1023418 INFO  (TEST-TriggerIntegrationTest.testSetProperties-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
   [junit4]   2> 1023418 DEBUG (zkCallback-1539-thread-1-processing-n:127.0.0.1:33721_solr) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 1023418 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 1023420 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1023420 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1023420 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 1024426 DEBUG (TEST-TriggerIntegrationTest.testSetProperties-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.a.ScheduledTriggers Changing value of autoscaling property: triggerScheduleDelaySeconds from: 1 to: 4
   [junit4]   2> 1032428 DEBUG (TEST-TriggerIntegrationTest.testSetProperties-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.a.ScheduledTriggers Changing value of autoscaling property: triggerScheduleDelaySeconds from: 4 to: 1
   [junit4]   2> 1042431 DEBUG (TEST-TriggerIntegrationTest.testSetProperties-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.a.ScheduledTriggers Changing value of autoscaling property: triggerCorePoolSize from: 4 to: 6
   [junit4]   2> 1052435 INFO  (TEST-TriggerIntegrationTest.testSetProperties-seed#[5BFC67CD2302993B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSetProperties
   [junit4]   2> 1052440 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEventQueue
   [junit4]   2> 1052440 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 3
   [junit4]   2> 1052441 DEBUG (zkCallback-1539-thread-1-processing-n:127.0.0.1:33721_solr) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 1052441 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 1052441 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1052441 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1052441 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 1052455 INFO  (qtp1188638725-7867) [n:127.0.0.1:33721_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1052459 INFO  (qtp1188638725-7867) [n:127.0.0.1:33721_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=4
   [junit4]   2> 1052464 INFO  (qtp1188638725-7860) [n:127.0.0.1:33721_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 1052475 INFO  (qtp1668135643-7872) [n:127.0.0.1:41887_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 1052475 DEBUG (qtp1188638725-7869) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1052476 DEBUG (zkCallback-1539-thread-1-processing-n:127.0.0.1:33721_solr) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 1052476 INFO  (qtp1188638725-7869) [n:127.0.0.1:33721_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=15
   [junit4]   2> 1052476 INFO  (zkCallback-1539-thread-1-processing-n:127.0.0.1:33721_solr) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.TriggerIntegrationTest TestEventQueueAction instantiated
   [junit4]   2> 1052476 DEBUG (zkCallback-1539-thread-1-processing-n:127.0.0.1:33721_solr) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:41887_solr, 127.0.0.1:33721_solr]
   [junit4]   2> 1052476 DEBUG (zkCallback-1539-thread-1-processing-n:127.0.0.1:33721_solr) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventQueueAction}], enabled=true}
   [junit4]   2> 1052476 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 1052477 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1052477 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1052477 DEBUG (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 1052478 DEBUG (ScheduledTrigger-3378-thread-2) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1052478 DEBUG (ScheduledTrigger-3378-thread-2) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1052479 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1052479 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7dd2a06f{/solr,null,AVAILABLE}
   [junit4]   2> 1052480 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@19ab9164{SSL,[ssl, http/1.1]}{127.0.0.1:33209}
   [junit4]   2> 1052480 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.Server Started @1054493ms
   [junit4]   2> 1052480 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33209}
   [junit4]   2> 1052480 ERROR (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1052480 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 1052480 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1052480 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1052480 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-03T01:50:26.282750Z
   [junit4]   2> 1052482 INFO  (zkConnectionManagerCallback-1554-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1052482 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1052487 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41913/solr
   [junit4]   2> 1052488 INFO  (zkConnectionManagerCallback-1558-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1052491 INFO  (zkConnectionManagerCallback-1560-thread-1-processing-n:127.0.0.1:33209_solr) [n:127.0.0.1:33209_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1052493 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [n:127.0.0.1:33209_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1052494 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [n:127.0.0.1:33209_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1052495 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [n:127.0.0.1:33209_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33209_solr
   [junit4]   2> 1052496 INFO  (zkCallback-1541-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1052496 INFO  (zkCallback-1539-thread-1-processing-n:127.0.0.1:33721_solr) [n:127.0.0.1:33721_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1052496 INFO  (zkCallback-1551-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1052496 INFO  (zkCallback-1559-thread-1-processing-n:127.0.0.1:33209_solr) [n:127.0.0.1:33209_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1052565 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [n:127.0.0.1:33209_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33209.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1052576 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [n:127.0.0.1:33209_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33209.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1052576 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [n:127.0.0.1:33209_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33209.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1052578 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [n:127.0.0.1:33209_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5BFC67CD2302993B-001/tempDir-001/node3/.
   [junit4]   2> 1053479 DEBUG (ScheduledTrigger-3378-thread-1) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1053479 DEBUG (ScheduledTrigger-3378-thread-1) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1053479 DEBUG (ScheduledTrigger-3378-thread-1) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:33209_solr at time 2645369253298307
   [junit4]   2> 1054457 INFO  (OverseerCollectionConfigSetProcessor-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1054479 DEBUG (ScheduledTrigger-3378-thread-1) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1054479 DEBUG (ScheduledTrigger-3378-thread-1) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1054479 DEBUG (ScheduledTrigger-3378-thread-1) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:33209_solr] added at times [2645369253298307], now=2645370253941001
   [junit4]   2> 1054480 DEBUG (ScheduledTrigger-3378-thread-1) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"965f306a48083T3i5888ncesuelojvumj3rluxt",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":2645369253298307,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2645369253298307],
   [junit4]   2>     "nodeNames":["127.0.0.1:33209_solr"]}}
   [junit4]   2> 1054480 INFO  (ScheduledTrigger-3378-thread-1) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1054480 DEBUG (AutoscalingActionExecutor-3379-thread-1-processing-n:127.0.0.1:33721_solr) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"965f306a48083T3i5888ncesuelojvumj3rluxt",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":2645369253298307,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2645369253298307],
   [junit4]   2>     "_enqueue_time_":2645370254183180,
   [junit4]   2>     "nodeNames":["127.0.0.1:33209_solr"]}}
   [junit4]   2> 1054480 INFO  (AutoscalingActionExecutor-3379-thread-1-processing-n:127.0.0.1:33721_solr) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.TriggerIntegrationTest -- event: {
   [junit4]   2>   "id":"965f306a48083T3i5888ncesuelojvumj3rluxt",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":2645369253298307,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2645369253298307],
   [junit4]   2>     "_enqueue_time_":2645370254183180,
   [junit4]   2>     "nodeNames":["127.0.0.1:33209_solr"]}}
   [junit4]   2> 1054481 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@6c7fa7d9{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1054481 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1348630078
   [junit4]   2> 1054482 DEBUG (AutoscalingActionExecutor-3379-thread-1-processing-n:127.0.0.1:33721_solr) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 1 ms for event id=965f306a48083T3i5888ncesuelojvumj3rluxt
   [junit4]   2> 1054486 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1054486 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1d1b4038: rootName = solr_33721, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@54213352
   [junit4]   2> 1054493 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1054493 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3f6e610e: rootName = solr_33721, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@422bfbc5
   [junit4]   2> 1054498 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1054499 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1718b44b: rootName = solr_33721, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@6d6d17e6
   [junit4]   2> 1054499 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:33721_solr
   [junit4]   2> 1054500 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1054500 INFO  (zkCallback-1559-thread-1-processing-n:127.0.0.1:33209_solr) [n:127.0.0.1:33209_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1054500 INFO  (zkCallback-1539-thread-1-processing-n:127.0.0.1:33721_solr) [n:127.0.0.1:33721_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1054501 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1054502 INFO  (zkCallback-1541-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1054502 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.Overseer Overseer (id=99283382892036101-127.0.0.1:33721_solr-n_0000000000) closing
   [junit4]   2> 1054502 INFO  (zkCallback-1551-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1054502 INFO  (OverseerStateUpdate-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:33721_solr
   [junit4]   2> 1054502 WARN  (OverseerAutoScalingTriggerThread-99283382892036101-127.0.0.1:33721_solr-n_0000000000) [n:127.0.0.1:33721_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1054502 DEBUG (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1054503 INFO  (zkCallback-1541-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41887_solr
   [junit4]   2> 1054503 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@43fd59d5{/solr,null,UNAVAILABLE}
   [junit4]   2> 1054503 INFO  (zkCallback-1541-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.Overseer Overseer (id=99283382892036102-127.0.0.1:41887_solr-n_0000000001) starting
   [junit4]   2> 1054505 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1054506 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 1054506 INFO  (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.TriggerIntegrationTest TestEventQueueAction instantiated
   [junit4]   2> 1054506 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:41887_solr, 127.0.0.1:33209_solr]
   [junit4]   2> 1054506 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventQueueAction}], enabled=true}
   [junit4]   2> 1054506 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:41887_solr, 127.0.0.1:33209_solr]
   [junit4]   2> 1054506 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion -1
   [junit4]   2> 1054506 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 1054507 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1054507 DEBUG (ScheduledTrigger-3395-thread-1) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1054507 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 1054507 DEBUG (ScheduledTrigger-3395-thread-2) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1054507 DEBUG (ScheduledTrigger-3395-thread-2) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1055507 DEBUG (ScheduledTrigger-3395-thread-1) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1055508 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1055508 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1056508 DEBUG (ScheduledTrigger-3395-thread-2) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1056508 DEBUG (ScheduledTrigger-3395-thread-4) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1056508 DEBUG (ScheduledTrigger-3395-thread-4) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1057508 DEBUG (ScheduledTrigger-3395-thread-1) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1057508 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1057508 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1058508 DEBUG (ScheduledTrigger-3395-thread-2) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1058508 DEBUG (ScheduledTrigger-3395-thread-4) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1058508 DEBUG (ScheduledTrigger-3395-thread-4) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1059508 DEBUG (ScheduledTrigger-3395-thread-1) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1059509 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1059509 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1060508 DEBUG (ScheduledTrigger-3395-thread-2) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1060509 DEBUG (ScheduledTrigger-3395-thread-4) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1060509 DEBUG (ScheduledTrigger-3395-thread-4) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1061509 DEBUG (ScheduledTrigger-3395-thread-1) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1061509 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1061509 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1062503 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[5BFC67CD2302993B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testEventQueue
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TriggerIntegrationTest -Dtests.method=testEventQueue -Dtests.seed=5BFC67CD2302993B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=kea -Dtests.timezone=America/Punta_Arenas -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 10.1s J0 | TriggerIntegrationTest.testEventQueue <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: action wasn't interrupted
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([5BFC67CD2302993B:924925632A655FCE]:0)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue(TriggerIntegrationTest.java:711)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> 1062509 DEBUG (ScheduledTrigger-3395-thread-2) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1062509 DEBUG (ScheduledTrigger-3395-thread-4) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1062509 DEBUG (ScheduledTrigger-3395-thread-4) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1062510 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEventFromRestoredState
   [junit4]   2> 1062510 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 6
   [junit4]   2> 1062510 DEBUG (zkCallback-1541-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 1062510 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 1062511 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1062511 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1062511 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 1062522 INFO  (qtp2065325771-7937) [n:127.0.0.1:33209_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 1062527 INFO  (qtp1668135643-7872) [n:127.0.0.1:41887_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 1062527 DEBUG (qtp2065325771-7930) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1062527 INFO  (qtp2065325771-7930) [n:127.0.0.1:33209_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=6
   [junit4]   2> 1062527 DEBUG (zkCallback-1541-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 1062527 DEBUG (zkCallback-1541-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:41887_solr, 127.0.0.1:33209_solr]
   [junit4]   2> 1062528 DEBUG (zkCallback-1541-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
   [junit4]   2> 1062528 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 1062528 INFO  (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1062528 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1062528 DEBUG (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 1062528 INFO  (qtp2065325771-7935) [n:127.0.0.1:33209_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1062528 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1062528 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1062531 INFO  (qtp2065325771-7935) [n:127.0.0.1:33209_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=3
   [junit4]   2> 1062532 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1062533 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3e4293b5{/solr,null,AVAILABLE}
   [junit4]   2> 1062533 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@33a943da{SSL,[ssl, http/1.1]}{127.0.0.1:37659}
   [junit4]   2> 1062534 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.Server Started @1064547ms
   [junit4]   2> 1062534 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37659}
   [junit4]   2> 1062534 ERROR (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1062534 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 1062534 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1062534 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1062534 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-03T01:50:36.336590Z
   [junit4]   2> 1062535 INFO  (zkConnectionManagerCallback-1564-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1062535 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1062540 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41913/solr
   [junit4]   2> 1062541 INFO  (zkConnectionManagerCallback-1568-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1062543 INFO  (zkConnectionManagerCallback-1570-thread-1-processing-n:127.0.0.1:37659_solr) [n:127.0.0.1:37659_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1062545 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [n:127.0.0.1:37659_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1062546 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [n:127.0.0.1:37659_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1062547 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [n:127.0.0.1:37659_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37659_solr
   [junit4]   2> 1062550 INFO  (zkCallback-1559-thread-1-processing-n:127.0.0.1:33209_solr) [n:127.0.0.1:33209_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1062550 INFO  (zkCallback-1541-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1062552 INFO  (zkCallback-1551-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1062552 INFO  (zkCallback-1569-thread-1-processing-n:127.0.0.1:37659_solr) [n:127.0.0.1:37659_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1062598 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [n:127.0.0.1:37659_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37659.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1062604 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [n:127.0.0.1:37659_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37659.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1062604 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [n:127.0.0.1:37659_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37659.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1062605 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [n:127.0.0.1:37659_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5BFC67CD2302993B-001/tempDir-001/node4/.
   [junit4]   2> 1063529 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1063529 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1063529 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:37659_solr at time 2645379303630387
   [junit4]   2> 1064530 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1064530 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1064530 INFO  (OverseerCollectionConfigSetProcessor-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1065530 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1065530 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1066530 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1066530 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1067530 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1067530 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1068530 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1068530 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1069531 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1069531 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1070531 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1070531 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1071531 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1071531 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1072531 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1072531 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1073531 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1073532 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1073532 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:37659_solr] added at times [2645379303630387], now=2645389306220844
   [junit4]   2> 1073532 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"965f55db06633T3i5888ncesuelojvumj3rluxv",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":2645379303630387,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2645379303630387],
   [junit4]   2>     "nodeNames":["127.0.0.1:37659_solr"]}}
   [junit4]   2> 1073532 INFO  (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1073532 DEBUG (AutoscalingActionExecutor-3396-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"965f55db06633T3i5888ncesuelojvumj3rluxv",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":2645379303630387,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2645379303630387],
   [junit4]   2>     "_enqueue_time_":2645389306370054,
   [junit4]   2>     "nodeNames":["127.0.0.1:37659_solr"]}}
   [junit4]   2> 1073533 DEBUG (AutoscalingActionExecutor-3396-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 0 ms for event id=965f55db06633T3i5888ncesuelojvumj3rluxv
   [junit4]   2> 1073534 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1073534 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6c339366{/solr,null,AVAILABLE}
   [junit4]   2> 1073535 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@11b055bf{SSL,[ssl, http/1.1]}{127.0.0.1:46833}
   [junit4]   2> 1073535 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.Server Started @1075549ms
   [junit4]   2> 1073535 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46833}
   [junit4]   2> 1073535 ERROR (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1073535 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 1073535 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1073535 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1073536 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-03T01:50:47.338Z
   [junit4]   2> 1073537 INFO  (zkConnectionManagerCallback-1574-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1073537 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1073544 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41913/solr
   [junit4]   2> 1073545 INFO  (zkConnectionManagerCallback-1578-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1073547 INFO  (zkConnectionManagerCallback-1580-thread-1-processing-n:127.0.0.1:46833_solr) [n:127.0.0.1:46833_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1073550 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [n:127.0.0.1:46833_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1073551 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [n:127.0.0.1:46833_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1073552 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [n:127.0.0.1:46833_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46833_solr
   [junit4]   2> 1073553 INFO  (zkCallback-1551-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1073553 INFO  (zkCallback-1559-thread-1-processing-n:127.0.0.1:33209_solr) [n:127.0.0.1:33209_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1073553 INFO  (zkCallback-1541-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1073553 INFO  (zkCallback-1569-thread-1-processing-n:127.0.0.1:37659_solr) [n:127.0.0.1:37659_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1073554 INFO  (zkCallback-1579-thread-1-processing-n:127.0.0.1:46833_solr) [n:127.0.0.1:46833_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1073586 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [n:127.0.0.1:46833_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46833.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1073593 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [n:127.0.0.1:46833_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46833.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1073593 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [n:127.0.0.1:46833_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46833.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d5e8331
   [junit4]   2> 1073594 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [n:127.0.0.1:46833_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5BFC67CD2302993B-001/tempDir-001/node5/.
   [junit4]   2> 1074533 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1074533 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1074533 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:46833_solr at time 2645390307773086
   [junit4]   2> 1075534 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1075534 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1076534 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1076534 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1077534 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1077534 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1078534 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1078534 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1079534 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1079535 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1080535 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1080535 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1081535 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1081535 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1082535 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1082535 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1083535 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1083535 DEBUG (ScheduledTrigger-3395-thread-3) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1083624 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@cb2b402{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1083624 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=753034772
   [junit4]   2> 1083624 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1083624 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@77cd0077: rootName = solr_41887, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@b310fe
   [junit4]   2> 1083632 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1083632 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@46487c16: rootName = solr_41887, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@422bfbc5
   [junit4]   2> 1083638 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1083638 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7f88c378: rootName = solr_41887, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@6d6d17e6
   [junit4]   2> 1083639 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:41887_solr
   [junit4]   2> 1083639 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1083639 INFO  (zkCallback-1551-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1083639 INFO  (zkCallback-1579-thread-1-processing-n:127.0.0.1:46833_solr) [n:127.0.0.1:46833_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1083639 INFO  (zkCallback-1541-thread-1-processing-n:127.0.0.1:41887_solr) [n:127.0.0.1:41887_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1083640 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1083639 INFO  (zkCallback-1569-thread-1-processing-n:127.0.0.1:37659_solr) [n:127.0.0.1:37659_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1083640 INFO  (zkCallback-1559-thread-1-processing-n:127.0.0.1:33209_solr) [n:127.0.0.1:33209_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1083640 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.Overseer Overseer (id=99283382892036102-127.0.0.1:41887_solr-n_0000000001) closing
   [junit4]   2> 1083640 INFO  (OverseerStateUpdate-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:41887_solr
   [junit4]   2> 1083640 WARN  (OverseerAutoScalingTriggerThread-99283382892036102-127.0.0.1:41887_solr-n_0000000001) [n:127.0.0.1:41887_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1083641 DEBUG (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1083641 INFO  (zkCallback-1559-thread-1-processing-n:127.0.0.1:33209_solr) [n:127.0.0.1:33209_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33209_solr
   [junit4]   2> 1083641 INFO  (zkCallback-1559-thread-1-processing-n:127.0.0.1:33209_solr) [n:127.0.0.1:33209_solr    ] o.a.s.c.Overseer Overseer (id=99283382892036107-127.0.0.1:33209_solr-n_0000000002) starting
   [junit4]   2> 1083642 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@53684d6a{/solr,null,UNAVAILABLE}
   [junit4]   2> 1083644 DEBUG (OverseerAutoScalingTriggerThread-99283382892036107-127.0.0.1:33209_solr-n_0000000002) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1083645 DEBUG (OverseerAutoScalingTriggerThread-99283382892036107-127.0.0.1:33209_solr-n_0000000002) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 1083645 DEBUG (OverseerAutoScalingTriggerThread-99283382892036107-127.0.0.1:33209_solr-n_0000000002) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:46833_solr, 127.0.0.1:37659_solr, 127.0.0.1:33209_solr]
   [junit4]   2> 1083645 DEBUG (OverseerAutoScalingTriggerThread-99283382892036107-127.0.0.1:33209_solr-n_0000000002) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
   [junit4]   2> 1083645 DEBUG (OverseerAutoScalingTriggerThread-99283382892036107-127.0.0.1:33209_solr-n_0000000002) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:46833_solr, 127.0.0.1:37659_solr, 127.0.0.1:33209_solr]
   [junit4]   2> 1083645 DEBUG (OverseerAutoScalingTriggerThread-99283382892036107-127.0.0.1:33209_solr-n_0000000002) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion -1
   [junit4]   2> 1083645 DEBUG (OverseerAutoScalingTriggerThread-99283382892036107-127.0.0.1:33209_solr-n_0000000002) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 1083646 INFO  (OverseerAutoScalingTriggerThread-99283382892036107-127.0.0.1:33209_solr-n_0000000002) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1083646 DEBUG (ScheduledTrigger-3412-thread-1) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1083646 DEBUG (OverseerAutoScalingTriggerThread-99283382892036107-127.0.0.1:33209_solr-n_0000000002) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.NodeAddedTrigger Adding node from marker path: 127.0.0.1:46833_solr
   [junit4]   2> 1083646 DEBUG (OverseerAutoScalingTriggerThread-99283382892036107-127.0.0.1:33209_solr-n_0000000002) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 1083646 DEBUG (ScheduledTrigger-3412-thread-1) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1083646 DEBUG (ScheduledTrigger-3412-thread-1) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1084646 DEBUG (ScheduledTrigger-3412-thread-2) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1084647 DEBUG (ScheduledTrigger-3412-thread-3) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1084647 DEBUG (ScheduledTrigger-3412-thread-3) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1084647 DEBUG (ScheduledTrigger-3412-thread-3) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:46833_solr] added at times [2645390307773086], now=2645400421494330
   [junit4]   2> 1084647 DEBUG (ScheduledTrigger-3412-thread-3) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"965f7ed964a9eT3i5888ncesuelojvumj3rluxx",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":2645390307773086,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2645390307773086],
   [junit4]   2>     "nodeNames":["127.0.0.1:46833_solr"]}}
   [junit4]   2> 1084648 INFO  (ScheduledTrigger-3412-thread-3) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1084648 DEBUG (AutoscalingActionExecutor-3413-thread-1-processing-n:127.0.0.1:33209_solr) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"965f7ed964a9eT3i5888ncesuelojvumj3rluxx",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":2645390307773086,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2645390307773086],
   [junit4]   2>     "_enqueue_time_":2645400421656753,
   [junit4]   2>     "nodeNames":["127.0.0.1:46833_solr"]}}
   [junit4]   2> 1084648 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[5BFC67CD2302993B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testEventFromRestoredState
   [junit4]   2> 1084649 DEBUG (AutoscalingActionExecutor-3413-thread-1-processing-n:127.0.0.1:33209_solr) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 0 ms for event id=965f7ed964a9eT3i5888ncesuelojvumj3rluxx
   [junit4]   2> 1084667 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[5BFC67CD2302993B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
   [junit4]   2> 1084668 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 9
   [junit4]   2> 1084668 DEBUG (zkCallback-1559-thread-1-processing-n:127.0.0.1:33209_solr) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 1084668 DEBUG (OverseerAutoScalingTriggerThread-99283382892036107-127.0.0.1:33209_solr-n_0000000002) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 1084669 DEBUG (OverseerAutoScalingTriggerThread-99283382892036107-127.0.0.1:33209_solr-n_0000000002) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1084669 DEBUG (OverseerAutoScalingTriggerThread-99283382892036107-127.0.0.1:33209_solr-n_0000000002) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1084669 DEBUG (OverseerAutoScalingTriggerThread-99283382892036107-127.0.0.1:33209_solr-n_0000000002) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 1084682 INFO  (qtp1431439142-7999) [n:127.0.0.1:46833_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 1084686 INFO  (qtp2065325771-7931) [n:127.0.0.1:33209_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 1084696 INFO  (qtp2022237916-7973) [n:127.0.0.1:37659_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 1084696 DEBUG (qtp2065325771-7936) [n:127.0.0.1:33209_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1084696 INFO  (qtp2065325771-7936) [n:127.0.0.1:33209_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime

[...truncated too long message...]

t4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: No registered leader was found after waiting for 1220000ms , collection: control_collection slice: shard1 saw state=null with live_nodes=[]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:771)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:736)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1178)
   [junit4]   2> 	... 41 more
   [junit4]   2> 
   [junit4]   2> 1577006 INFO  (qtp657625067-11979) [n:127.0.0.1:44667_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=630168
   [junit4]   2> 1577006 ERROR (TEST-SolrShardReporterTest.test-seed#[5BFC67CD2302993B]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1577007 INFO  (TEST-SolrShardReporterTest.test-seed#[5BFC67CD2302993B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:32875 32875
   [junit4]   2> 1582404 INFO  (Thread-3887) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:32875 32875
   [junit4]   2> 1582404 WARN  (Thread-3887) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/aliases.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/clusterprops.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/live_nodes
   [junit4]   2> 	2	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SolrShardReporterTest -Dtests.method=test -Dtests.seed=5BFC67CD2302993B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=rw-RW -Dtests.timezone=CTT -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR    636s J2 | SolrShardReporterTest.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:44667
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([5BFC67CD2302993B:D3A858178DFEF4C3]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:320)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
   [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > 	at java.base/java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > 	at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]    > 	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]    > 	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]    > 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]    > 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]    > 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]    > 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]    > 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]    > 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
   [junit4]    > 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]    > 	... 42 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.metrics.reporters.solr.SolrShardReporterTest_5BFC67CD2302993B-001
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=false): {}, locale=rw-RW, timezone=CTT
   [junit4]   2> NOTE: Linux 4.10.0-40-generic amd64/Oracle Corporation 9.0.1 (64-bit)/cpus=8,threads=1,free=325032040,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestPullReplica, SpellPossibilityIteratorTest, BasicDistributedZk2Test, MetricsHandlerTest, FileBasedSpellCheckerTest, TestMaxTokenLenTokenizer, TestNoOpRegenerator, TestChildDocTransformer, TestDistributedMissingSort, RecoveryZkTest, IndexSchemaTest, TestUninvertingReader, TestSolrCLIRunExample, TestFieldResource, DistributedQueryComponentOptimizationTest, TestCharFilters, LeaderElectionTest, CopyFieldTest, DeleteNodeTest, TestLegacyFieldCache, BadCopyFieldTest, TestManagedStopFilterFactory, TestMaxScoreQueryParser, RuleEngineTest, TestFunctionQuery, TestRestManager, TestHdfsCloudBackupRestore, TestExportWriter, DirectSolrConnectionTest, TestSolrXml, TestNestedDocsSort, TestFieldSortValues, SolrJmxReporterTest, RecoveryAfterSoftCommitTest, DocValuesNotIndexedTest, XsltUpdateRequestHandlerTest, InfixSuggestersTest, WordBreakSolrSpellCheckerTest, AutoAddReplicasIntegrationTest, TestCloudSearcherWarming, UpdateLogTest, TestPivotHelperCode, TestLRUStatsCache, TestCloudPivotFacet, ZkNodePropsTest, HdfsThreadLeakTest, TestSmileRequest, StatsComponentTest, SuggesterTest, TestDynamicLoading, TestDistributedStatsComponentCardinality, DistributedFacetPivotSmallTest, ComputePlanActionTest, TestEmbeddedSolrServerConstructors, TestTrie, UninvertDocValuesMergePolicyTest, TestMinMaxOnMultiValuedField, EnumFieldTest, SolrGangliaReporterTest, TestCoreContainer, MoveReplicaHDFSTest, TestComputePlanAction, CurrencyFieldTypeTest, ZkStateReaderTest, TestDistribIDF, LeaderElectionContextKeyTest, XmlUpdateRequestHandlerTest, DirectSolrSpellCheckerTest, SystemInfoHandlerTest, TestLMJelinekMercerSimilarityFactory, TestMacroExpander, IndexBasedSpellCheckerTest, TestSQLHandler, TestConfigSets, JvmMetricsTest, DistributedDebugComponentTest, SolrShardReporterTest]
   [junit4] Completed [332/764 (2!)] on J2 in 635.99s, 1 test, 1 error <<< FAILURES!

[...truncated 42287 lines...]

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk1.8.0_144) - Build # 1105 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1105/
Java: 64bit/jdk1.8.0_144 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:     1) Thread[id=12549, name=jetty-launcher-5501-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)    2) Thread[id=12559, name=jetty-launcher-5501-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   1) Thread[id=12549, name=jetty-launcher-5501-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   2) Thread[id=12559, name=jetty-launcher-5501-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
	at __randomizedtesting.SeedInfo.seed([2E4EF436C9211EC0]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.security.BasicAuthIntegrationTest

Error Message:
Error from server at http://127.0.0.1:37501/solr: create the collection time out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:37501/solr: create the collection time out:180s
	at __randomizedtesting.SeedInfo.seed([2E4EF436C9211EC0]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.security.BasicAuthIntegrationTest.setupCluster(BasicAuthIntegrationTest.java:82)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ShardSplitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:44023

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:44023
	at __randomizedtesting.SeedInfo.seed([2E4EF436C9211EC0:A61ACBEC67DD7338]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:928)
	at org.apache.solr.cloud.ShardSplitTest.splitByRouteKeyTest(ShardSplitTest.java:797)
	at org.apache.solr.cloud.ShardSplitTest.test(ShardSplitTest.java:102)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 47 more


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestComputePlanAction.testNodeAdded

Error Message:


Stack Trace:
java.util.ConcurrentModificationException
	at __randomizedtesting.SeedInfo.seed([2E4EF436C9211EC0:4B8DA2416B82B6C3]:0)
	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
	at java.util.ArrayList$Itr.next(ArrayList.java:851)
	at org.apache.solr.cloud.autoscaling.sim.SimSolrCloudTestCase.tearDown(SimSolrCloudTestCase.java:141)
	at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12134 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
   [junit4]   2> 585406 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[2E4EF436C9211EC0]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_2E4EF436C9211EC0-001/init-core-data-001
   [junit4]   2> 585407 WARN  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[2E4EF436C9211EC0]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=10 numCloses=10
   [junit4]   2> 585407 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[2E4EF436C9211EC0]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 585408 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[2E4EF436C9211EC0]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 585422 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[2E4EF436C9211EC0]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_2E4EF436C9211EC0-001/tempDir-001
   [junit4]   2> 585422 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[2E4EF436C9211EC0]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 585422 INFO  (Thread-987) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 585422 INFO  (Thread-987) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 585425 ERROR (Thread-987) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 585522 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[2E4EF436C9211EC0]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:40135
   [junit4]   2> 585524 INFO  (zkConnectionManagerCallback-5504-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 585526 INFO  (jetty-launcher-5501-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 585526 INFO  (jetty-launcher-5501-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 585528 INFO  (jetty-launcher-5501-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@433ad9a1{/solr,null,AVAILABLE}
   [junit4]   2> 585528 INFO  (jetty-launcher-5501-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6a412ee3{/solr,null,AVAILABLE}
   [junit4]   2> 585529 INFO  (jetty-launcher-5501-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5a071ef{SSL,[ssl, http/1.1]}{127.0.0.1:46883}
   [junit4]   2> 585529 INFO  (jetty-launcher-5501-thread-1) [    ] o.e.j.s.Server Started @587585ms
   [junit4]   2> 585529 INFO  (jetty-launcher-5501-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46883}
   [junit4]   2> 585529 ERROR (jetty-launcher-5501-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 585529 INFO  (jetty-launcher-5501-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 585529 INFO  (jetty-launcher-5501-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 585529 INFO  (jetty-launcher-5501-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 585529 INFO  (jetty-launcher-5501-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-02T22:25:30.659Z
   [junit4]   2> 585530 INFO  (jetty-launcher-5501-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7e4afb14{SSL,[ssl, http/1.1]}{127.0.0.1:37507}
   [junit4]   2> 585530 INFO  (jetty-launcher-5501-thread-2) [    ] o.e.j.s.Server Started @587587ms
   [junit4]   2> 585530 INFO  (jetty-launcher-5501-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37507}
   [junit4]   2> 585530 ERROR (jetty-launcher-5501-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 585531 INFO  (jetty-launcher-5501-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 585531 INFO  (jetty-launcher-5501-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 585531 INFO  (jetty-launcher-5501-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 585531 INFO  (jetty-launcher-5501-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-02T22:25:30.661Z
   [junit4]   2> 585532 INFO  (zkConnectionManagerCallback-5506-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 585532 INFO  (jetty-launcher-5501-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 585532 INFO  (zkConnectionManagerCallback-5508-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 585536 INFO  (jetty-launcher-5501-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 585540 INFO  (jetty-launcher-5501-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40135/solr
   [junit4]   2> 585542 INFO  (jetty-launcher-5501-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40135/solr
   [junit4]   2> 585543 INFO  (zkConnectionManagerCallback-5512-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 585546 INFO  (zkConnectionManagerCallback-5516-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 585548 INFO  (zkConnectionManagerCallback-5519-thread-1-processing-n:127.0.0.1:37507_solr) [n:127.0.0.1:37507_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 585554 INFO  (zkConnectionManagerCallback-5520-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 585588 INFO  (jetty-launcher-5501-thread-2) [n:127.0.0.1:37507_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 585588 INFO  (jetty-launcher-5501-thread-2) [n:127.0.0.1:37507_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37507_solr
   [junit4]   2> 585589 INFO  (jetty-launcher-5501-thread-2) [n:127.0.0.1:37507_solr    ] o.a.s.c.Overseer Overseer (id=99282579000721413-127.0.0.1:37507_solr-n_0000000000) starting
   [junit4]   2> 585592 INFO  (jetty-launcher-5501-thread-2) [n:127.0.0.1:37507_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37507_solr
   [junit4]   2> 585592 INFO  (OverseerStateUpdate-99282579000721413-127.0.0.1:37507_solr-n_0000000000) [n:127.0.0.1:37507_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 585594 INFO  (jetty-launcher-5501-thread-1) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 585594 INFO  (jetty-launcher-5501-thread-1) [n:127.0.0.1:46883_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 585595 INFO  (jetty-launcher-5501-thread-1) [n:127.0.0.1:46883_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46883_solr
   [junit4]   2> 585602 INFO  (zkCallback-5518-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 585602 INFO  (zkCallback-5517-thread-2-processing-n:127.0.0.1:37507_solr) [n:127.0.0.1:37507_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 585629 INFO  (jetty-launcher-5501-thread-1) [n:127.0.0.1:46883_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
   [junit4]   2> 585632 WARN  (jetty-launcher-5501-thread-1) [n:127.0.0.1:46883_solr    ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
   [junit4]   2> 585633 INFO  (jetty-launcher-5501-thread-1) [n:127.0.0.1:46883_solr    ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=localhost.localdomain,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=jenkins, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
   [junit4]   2> 585635 INFO  (jetty-launcher-5501-thread-1) [n:127.0.0.1:46883_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 585645 INFO  (jetty-launcher-5501-thread-1-EventThread) [n:127.0.0.1:46883_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 585656 WARN  (jetty-launcher-5501-thread-1) [n:127.0.0.1:46883_solr    ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
   [junit4]   2> 585656 INFO  (jetty-launcher-5501-thread-2) [n:127.0.0.1:37507_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
   [junit4]   2> 585657 WARN  (jetty-launcher-5501-thread-2) [n:127.0.0.1:37507_solr    ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
   [junit4]   2> 585658 INFO  (jetty-launcher-5501-thread-2) [n:127.0.0.1:37507_solr    ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=localhost.localdomain,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=jenkins, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
   [junit4]   2> 585658 INFO  (jetty-launcher-5501-thread-2) [n:127.0.0.1:37507_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 585660 INFO  (jetty-launcher-5501-thread-2-EventThread) [n:127.0.0.1:37507_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 585665 WARN  (jetty-launcher-5501-thread-2) [n:127.0.0.1:37507_solr    ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
   [junit4]   2> 585693 INFO  (jetty-launcher-5501-thread-2) [n:127.0.0.1:37507_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37507.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78c6e655
   [junit4]   2> 585695 INFO  (jetty-launcher-5501-thread-1) [n:127.0.0.1:46883_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46883.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78c6e655
   [junit4]   2> 585700 INFO  (jetty-launcher-5501-thread-1) [n:127.0.0.1:46883_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46883.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78c6e655
   [junit4]   2> 585700 INFO  (jetty-launcher-5501-thread-1) [n:127.0.0.1:46883_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46883.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78c6e655
   [junit4]   2> 585701 INFO  (jetty-launcher-5501-thread-1) [n:127.0.0.1:46883_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_2E4EF436C9211EC0-001/tempDir-001/node1/.
   [junit4]   2> 585702 INFO  (jetty-launcher-5501-thread-2) [n:127.0.0.1:37507_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37507.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78c6e655
   [junit4]   2> 585702 INFO  (jetty-launcher-5501-thread-2) [n:127.0.0.1:37507_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37507.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78c6e655
   [junit4]   2> 585703 INFO  (jetty-launcher-5501-thread-2) [n:127.0.0.1:37507_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_2E4EF436C9211EC0-001/tempDir-001/node2/.
   [junit4]   2> 585760 INFO  (zkConnectionManagerCallback-5526-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 585779 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigGroups
   [junit4]   2> 585805 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[2E4EF436C9211EC0]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=noGroups&p=noGroups&t=test&e=1514967930930&s=D2eY7mrbTH1XxcMm8NDWGzilqZM="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:30 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:30 GMT
   [junit4]   2> 585806 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigGroups
   [junit4]   2> 585814 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidRemote
   [junit4]   2> 585821 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[2E4EF436C9211EC0]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=localHostAnyGroup&p=localHostAnyGroup&t=test&e=1514967930945&s=dLn2FT8SNCZkIzVh4bwBRojnOek="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:30 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:30 GMT
   [junit4]   2> 585822 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidRemote
   [junit4]   2> 585832 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigHosts
   [junit4]   2> 585836 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[2E4EF436C9211EC0]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=noHosts&p=noHosts&t=test&e=1514967930963&s=46AW1dbxUYD1PsAkjM6ZDgsdRAg="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:30 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:30 GMT
   [junit4]   2> 585836 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigHosts
   [junit4]   2> 585848 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateGroup
   [junit4]   2> 585875 INFO  (qtp1632468781-12511) [n:127.0.0.1:46883_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=jenkins&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 585875 INFO  (qtp1632468781-12511) [n:127.0.0.1:46883_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=jenkins&action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 585876 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[2E4EF436C9211EC0]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=anyHostUsersGroup&p=anyHostUsersGroup&t=test&e=1514967930978&s=70lfmgdyYfNe8WPjlEzzyuT0lwg="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:30 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:30 GMT
   [junit4]   2> 585876 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateGroup
   [junit4]   2> 585886 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyUnknownRemote
   [junit4]   2> 587664 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[2E4EF436C9211EC0]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=localHostAnyGroup&p=localHostAnyGroup&t=test&e=1514967931017&s=519QxsaeRTWxCiFqarUtnA1cDxs="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:31 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:31 GMT
   [junit4]   2> 587665 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyUnknownRemote
   [junit4]   2> 587688 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateHost
   [junit4]   2> 587690 INFO  (qtp1632468781-12507) [n:127.0.0.1:46883_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 587690 INFO  (qtp1632468781-12507) [n:127.0.0.1:46883_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 587691 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[2E4EF436C9211EC0]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=localHostAnyGroup&p=localHostAnyGroup&t=test&e=1514967932819&s=F0ZZsXhX7ZKG15i/9d1el94UdBQ="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:32 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:32 GMT
   [junit4]   2> 587691 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateHost
   [junit4]   2> 587718 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNullProxyUser
   [junit4]   2> 587718 INFO  (qtp1632468781-12509) [    ] o.a.s.s.SolrDispatchFilter Error authenticating
   [junit4]   2> java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
   [junit4]   2> 	at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:134)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 587719 WARN  (qtp1632468781-12509) [    ] o.e.j.s.ServletHandler /solr/admin/collections
   [junit4]   2> org.apache.solr.common.SolrException: Error during request authentication, 
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:457)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
   [junit4]   2> 	at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:134)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
   [junit4]   2> 	... 29 more
   [junit4]   2> 587721 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNullProxyUser
   [junit4]   2> 587741 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testForwarding
   [junit4]   2> 587743 INFO  (zkConnectionManagerCallback-5529-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 587948 INFO  (zkConnectionManagerCallback-5532-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 587948 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[2E4EF436C9211EC0]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 587949 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[2E4EF436C9211EC0]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40135/solr ready
   [junit4]   2> 587953 INFO  (qtp1632468781-12509) [n:127.0.0.1:46883_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=1&collection.configName=conf1&maxShardsPerNode=1&name=forwardingCollection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 587955 INFO  (OverseerThreadFactory-2324-thread-1-processing-n:127.0.0.1:37507_solr) [n:127.0.0.1:37507_solr    ] o.a.s.c.CreateCollectionCmd Create collection forwardingCollection
   [junit4]   2> 588062 INFO  (OverseerStateUpdate-99282579000721413-127.0.0.1:37507_solr-n_0000000000) [n:127.0.0.1:37507_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"forwardingCollection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"forwardingCollection_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:46883/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 588266 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf1&newCollection=true&name=forwardingCollection_shard1_replica_n1&action=CREATE&numShards=1&collection=forwardingCollection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 588266 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 588369 INFO  (zkCallback-5518-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 588369 INFO  (zkCallback-5518-thread-2-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 589287 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 589309 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.IndexSchema [forwardingCollection_shard1_replica_n1] Schema name=test
   [junit4]   2> 589357 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 589364 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'forwardingCollection_shard1_replica_n1' using configuration from collection forwardingCollection, trusted=true
   [junit4]   2> 589364 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46883.solr.core.forwardingCollection.shard1.replica_n1' (registry 'solr.core.forwardingCollection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78c6e655
   [junit4]   2> 589364 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 589364 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [[forwardingCollection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_2E4EF436C9211EC0-001/tempDir-001/node1/forwardingCollection_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_2E4EF436C9211EC0-001/tempDir-001/node1/./forwardingCollection_shard1_replica_n1/data/]
   [junit4]   2> 589365 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=42, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=42.7939453125, floorSegmentMB=2.171875, forceMergeDeletesPctAllowed=0.25302482065251874, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 589393 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 589393 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 589394 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 589394 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 589394 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=19.4228515625, floorSegmentMB=1.9599609375, forceMergeDeletesPctAllowed=0.6658731617218094, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5246925338935616
   [junit4]   2> 589395 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3da086ba[forwardingCollection_shard1_replica_n1] main]
   [junit4]   2> 589395 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 589421 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 589512 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 589513 INFO  (searcherExecutor-2329-thread-1-processing-n:127.0.0.1:46883_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 589513 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588521268300218368
   [junit4]   2> 589513 INFO  (searcherExecutor-2329-thread-1-processing-n:127.0.0.1:46883_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 589513 INFO  (searcherExecutor-2329-thread-1-processing-n:127.0.0.1:46883_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 589513 INFO  (searcherExecutor-2329-thread-1-processing-n:127.0.0.1:46883_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 589513 INFO  (searcherExecutor-2329-thread-1-processing-n:127.0.0.1:46883_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 589513 INFO  (searcherExecutor-2329-thread-1-processing-n:127.0.0.1:46883_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 589514 INFO  (searcherExecutor-2329-thread-1-processing-n:127.0.0.1:46883_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 589514 INFO  (searcherExecutor-2329-thread-1-processing-n:127.0.0.1:46883_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 589514 INFO  (searcherExecutor-2329-thread-1-processing-n:127.0.0.1:46883_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 589516 INFO  (searcherExecutor-2329-thread-1-processing-n:127.0.0.1:46883_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [forwardingCollection_shard1_replica_n1] Registered new searcher Searcher@3da086ba[forwardingCollection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 589518 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 589518 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 589518 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:46883/solr/forwardingCollection_shard1_replica_n1/
   [junit4]   2> 589518 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 589518 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:46883/solr/forwardingCollection_shard1_replica_n1/ has no replicas
   [junit4]   2> 589518 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 589520 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:46883/solr/forwardingCollection_shard1_replica_n1/ shard1
   [junit4]   2> 589621 INFO  (zkCallback-5518-thread-2-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 589621 INFO  (zkCallback-5518-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 589670 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 589676 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf1&newCollection=true&name=forwardingCollection_shard1_replica_n1&action=CREATE&numShards=1&collection=forwardingCollection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1410
   [junit4]   2> 589677 WARN  (httpShardExecutor-2320-thread-1-processing-n:127.0.0.1:37507_solr [https:////127.0.0.1:46883//solr] https:////127.0.0.1:46883//solr) [n:127.0.0.1:37507_solr    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=$&p=$&t=test&e=1514967933395&s=zQl3UIAK0JQtjqR+OdQraDjUIe4="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:33 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:33 GMT
   [junit4]   2> 589681 INFO  (qtp1632468781-12509) [n:127.0.0.1:46883_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 589774 INFO  (zkCallback-5518-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 589774 INFO  (zkCallback-5518-thread-2-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 589957 INFO  (OverseerCollectionConfigSetProcessor-99282579000721413-127.0.0.1:37507_solr-n_0000000000) [n:127.0.0.1:37507_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 590681 INFO  (qtp1632468781-12509) [n:127.0.0.1:46883_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=1&collection.configName=conf1&maxShardsPerNode=1&name=forwardingCollection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2} status=0 QTime=2727
   [junit4]   2> 590681 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[2E4EF436C9211EC0]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=user&p=user&t=test&e=1514967933083&s=zZAmr0mrPwob619R0tcgy/soBm4="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:33 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:33 GMT
   [junit4]   2> 590682 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[2E4EF436C9211EC0]) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: forwardingCollection failOnTimeout:true timeout (sec):100
   [junit4]   2> 590682 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[2E4EF436C9211EC0]) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: forwardingCollection
   [junit4]   2> 590688 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.S.Request [forwardingCollection_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 590688 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[2E4EF436C9211EC0]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=user&p=user&t=test&e=1514967935817&s=Ne8pp5JsiPys/ab0JGZckqj8I+8="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:35 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:35 GMT
   [junit4]   2> 590725 INFO  (qtp1856911085-12499) [n:127.0.0.1:37507_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 590729 INFO  (qtp1632468781-12504) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.S.Request [forwardingCollection_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 590730 WARN  (qtp1856911085-12499) [n:127.0.0.1:37507_solr    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=$&p=$&t=test&e=1514967935858&s=PKOFWb4rw3LzKjjoVHyUqV38k+k="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:35 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:35 GMT
   [junit4]   2> 590730 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[2E4EF436C9211EC0]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=user&p=user&t=test&e=1514967935854&s=rSt+JSIlGO/jC9XMBszhP/9AkRE="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:35 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:35 GMT
   [junit4]   2> 590731 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[2E4EF436C9211EC0]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=$&p=$&t=test&e=1514967935858&s=PKOFWb4rw3LzKjjoVHyUqV38k+k="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:35 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:35 GMT
   [junit4]   2> 590731 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testForwarding
   [junit4]   2> 590759 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyWrongHost
   [junit4]   2> 600871 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[2E4EF436C9211EC0]-SendThread(127.0.0.1:40135)) [    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10189ms for sessionid 0x160b8fac5b9000b
   [junit4]   2> 600872 WARN  (jetty-launcher-5501-thread-2-SendThread(127.0.0.1:40135)) [n:127.0.0.1:37507_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15193ms for sessionid 0x160b8fac5b90008
   [junit4]   2> 600873 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x160b8fac5b90008, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 600873 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[2E4EF436C9211EC0]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=wrongHost&p=wrongHost&t=test&e=1514967935892&s=tMAwArjfy8mcVFVcIIulEUVNQMc="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:35 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:35 GMT
   [junit4]   2> 600873 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyWrongHost
   [junit4]   2> 600873 WARN  (jetty-launcher-5501-thread-1-SendThread(127.0.0.1:40135)) [n:127.0.0.1:46883_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15194ms for sessionid 0x160b8fac5b90007
   [junit4]   2> 600874 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x160b8fac5b90007, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 600888 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidProxyUser
   [junit4]   2> 600890 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[2E4EF436C9211EC0]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=bar&p=bar&t=test&e=1514967946019&s=fVbmTNa1y6GKucLP97+nA09V0qY="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:46 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:46 GMT
   [junit4]   2> 600891 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidProxyUser
   [junit4]   2> 600901 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateAnyHostAnyUser
   [junit4]   2> 600902 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 600902 INFO  (qtp1632468781-12502) [n:127.0.0.1:46883_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 600902 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[2E4EF436C9211EC0]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=anyHostAnyUser&p=anyHostAnyUser&t=test&e=1514967946031&s=2WTt1jU1xbrEkIfg/C8m/UMZOlk="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:46 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:46 GMT
   [junit4]   2> 600902 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateAnyHostAnyUser
   [junit4]   2> 600918 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidGroup
   [junit4]   2> 600924 WARN  (qtp1632468781-12509) [n:127.0.0.1:46883_solr    ] o.a.h.s.ShellBasedUnixGroupsMapping got exception trying to get groups for user bar: id: ‘bar’: no such user
   [junit4]   2> id: ‘bar’: no such user
   [junit4]   2> 
   [junit4]   2> 600925 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[2E4EF436C9211EC0]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=bogusGroup&p=bogusGroup&t=test&e=1514967946049&s=bqp4YI/95EfXplYOqNh3INFAKdw="; Path=/; Domain=127.0.0.1; Expires= , 03-   -2018 08:25:46 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 03- -2018 08:25:46 GMT
   [junit4]   2> 600925 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[2E4EF436C9211EC0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidGroup
   [junit4]   2> 600973 INFO  (jetty-closer-5502-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@7e4afb14{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 600973 INFO  (jetty-launcher-5501-thread-2-EventThread) [n:127.0.0.1:37507_solr    ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
   [junit4]   2> 600973 INFO  (jetty-closer-5502-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@5a071ef{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 600975 INFO  (jetty-closer-5502-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1216129654
   [junit4]   2> 600975 INFO  (jetty-closer-5502-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=708791087
   [junit4]   2> 600975 INFO  (jetty-closer-5502-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 600975 INFO  (jetty-closer-5502-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 600975 INFO  (jetty-closer-5502-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3bb73525: rootName = solr_37507, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@413d1b86
   [junit4]   2> 600977 INFO  (jetty-closer-5502-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@741bc4b5: rootName = solr_46883, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@3a660805
   [junit4]   2> 600979 INFO  (jetty-closer-5502-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 600979 INFO  (jetty-closer-5502-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4a449768: rootName = solr_37507, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@2ef0c827
   [junit4]   2> 600981 INFO  (jetty-closer-5502-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 600981 INFO  (jetty-closer-5502-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6fe6946e: rootName = solr_37507, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@2acf5921
   [junit4]   2> 600982 INFO  (jetty-closer-5502-thread-2) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:37507_solr
   [junit4]   2> 600982 INFO  (jetty-closer-5502-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 600982 INFO  (jetty-closer-5502-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@73ed9f68: rootName = solr_46883, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@2ef0c827
   [junit4]   2> 600982 INFO  (jetty-closer-5502-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 600982 INFO  (zkCallback-5518-thread-2-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 600982 INFO  (zkCallback-5517-thread-2-processing-n:127.0.0.1:37507_solr) [n:127.0.0.1:37507_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 600983 INFO  (jetty-closer-5502-thread-2) [    ] o.a.s.c.Overseer Overseer (id=99282579000721413-127.0.0.1:37507_solr-n_0000000000) closing
   [junit4]   2> 600983 INFO  (OverseerStateUpdate-99282579000721413-127.0.0.1:37507_solr-n_0000000000) [n:127.0.0.1:37507_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37507_solr
   [junit4]   2> 600983 WARN  (OverseerAutoScalingTriggerThread-99282579000721413-127.0.0.1:37507_solr-n_0000000000) [n:127.0.0.1:37507_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 600985 INFO  (zkCallback-5518-thread-2-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46883_solr
   [junit4]   2> 600985 ERROR (Thread[Thread-1006,5,TGRP-TestSolrCloudWithSecureImpersonation]) [n:127.0.0.1:37507_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 600986 INFO  (jetty-closer-5502-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 600986 INFO  (jetty-closer-5502-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@24bb5c60: rootName = solr_46883, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@2acf5921
   [junit4]   2> 600987 INFO  (jetty-closer-5502-thread-1) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:46883_solr
   [junit4]   2> 600987 INFO  (jetty-closer-5502-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 600990 INFO  (coreCloseExecutor-2335-thread-1) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [forwardingCollection_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4c757d00
   [junit4]   2> 600990 INFO  (coreCloseExecutor-2335-thread-1) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.forwardingCollection.shard1.replica_n1, tag=1282768128
   [junit4]   2> 600990 INFO  (coreCloseExecutor-2335-thread-1) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@63be63ac: rootName = solr_46883, domain = solr.core.forwardingCollection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.forwardingCollection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@37425e66
   [junit4]   2> 600995 INFO  (zkCallback-5518-thread-2-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 600998 INFO  (coreCloseExecutor-2335-thread-1) [n:127.0.0.1:46883_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.forwardingCollection.shard1.leader, tag=1282768128
   [junit4]   2> 601000 ERROR (Thread[Thread-1004,5,TGRP-TestSolrCloudWithSecureImpersonation]) [n:127.0.0.1:46883_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 601653 INFO  (jetty-closer-5502-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6a412ee3{/solr,null,UNAVAILABLE}
   [junit4]   2> 601993 INFO  (jetty-closer-5502-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@433ad9a1{/solr,null,UNAVAILABLE}
   [junit4]   2> 601993 ERROR (SUITE-TestSolrCloudWithSecureImpersonation-seed#[2E4EF436C9211EC0]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 601993 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[2E4EF436C9211EC0]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40135 40135
   [junit4]   2> 604611 INFO  (Thread-987) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40135 40135
   [junit4]   2> 604612 WARN  (Thread-987) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/aliases.json
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/collections/forwardingCollection/state.json
   [junit4]   2> 	3	/solr/clusterstate.json
   [junit4]   2> 	3	/solr/clusterprops.json
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 	3	/solr/collections
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
   [junit4]   2> 
   [junit4]   2> 604619 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[2E4EF436C9211EC0]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Jan 02, 2018 10:25:49 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> Jan 02, 2018 10:26:09 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   [junit4]   2>    1) Thread[id=12549, name=jetty-launcher-5501-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]   

[...truncated too long message...]

oncurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/live_nodes
   [junit4]   2> 	4	/solr/collections
   [junit4]   2> 
   [junit4]   2> 2971868 INFO  (SUITE-BasicAuthIntegrationTest-seed#[2E4EF436C9211EC0]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1210, maxMBSortInHeap=7.20103292503402, sim=RandomSimilarity(queryNorm=false): {}, locale=ko-KR, timezone=America/Jamaica
   [junit4]   2> NOTE: Linux 4.10.0-40-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=8,threads=1,free=274070336,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [V2StandaloneTest, FieldAnalysisRequestHandlerTest, SharedFSAutoReplicaFailoverTest, TestRestManager, StatsComponentTest, TestFieldSortValues, BJQParserTest, TestImpersonationWithHadoopAuth, TestNestedDocsSort, XsltUpdateRequestHandlerTest, UUIDUpdateProcessorFallbackTest, BlockCacheTest, TestEmbeddedSolrServerConstructors, JvmMetricsTest, LoggingHandlerTest, TestSubQueryTransformerDistrib, TestDistributedStatsComponentCardinality, TestCoreContainer, TestLMDirichletSimilarityFactory, TestFieldCache, LeaderElectionContextKeyTest, TestStressLiveNodes, TestDistributedGrouping, TestFileDictionaryLookup, TestMacroExpander, TestRuleBasedAuthorizationPlugin, TestHashPartitioner, OverseerTest, TestLegacyFieldCache, UninvertDocValuesMergePolicyTest, DocExpirationUpdateProcessorFactoryTest, CoreSorterTest, ScriptEngineTest, ConnectionReuseTest, TestCloudSearcherWarming, DistributedFacetSimpleRefinementLongTailTest, TestGenericDistributedQueue, TestLegacyTerms, TestCloudManagedSchema, CustomCollectionTest, HttpTriggerListenerTest, AliasIntegrationTest, TestFieldCacheSortRandom, TestSystemIdResolver, TestMinMaxOnMultiValuedField, TestGroupingSearch, TestCloudPivotFacet, TestSolrCLIRunExample, TestZkChroot, AnalyticsMergeStrategyTest, ReplicationFactorTest, TestPerFieldSimilarityWithDefaultOverride, MetricsHandlerTest, ComputePlanActionTest, TestSolrCloudWithSecureImpersonation, TestLeaderInitiatedRecoveryThread, TestTolerantSearch, TestCoreDiscovery, TestComputePlanAction, QueryElevationComponentTest, TestLMJelinekMercerSimilarityFactory, TestQuerySenderListener, CollectionsAPIAsyncDistributedZkTest, EchoParamsTest, WordBreakSolrSpellCheckerTest, BitVectorTest, TestJsonRequest, HighlighterTest, RuleEngineTest, TestPullReplica, UnloadDistributedZkTest, TestUninvertingReader, AtomicUpdateProcessorFactoryTest, CoreAdminHandlerTest, TestChildDocTransformer, BadCopyFieldTest, SmileWriterTest, TestNamedUpdateProcessors, TestValueSourceCache, HdfsBasicDistributedZkTest, TestRealTimeGet, MoveReplicaHDFSTest, TestTrieFacet, TestNoOpRegenerator, DirectSolrConnectionTest, TestElisionMultitermQuery, ConfigSetsAPITest, TestConfigReload, ZkStateReaderTest, SolrJmxReporterTest, TestManagedStopFilterFactory, DistribJoinFromCollectionTest, TestCollectionAPIs, CSVRequestHandlerTest, TestManagedSchemaThreadSafety, TestHalfAndHalfDocValues, BasicDistributedZk2Test, TestFuzzyAnalyzedSuggestions, TestShardHandlerFactory, DistributedQueryComponentOptimizationTest, DistributedSpellCheckComponentTest, SpatialHeatmapFacetsTest, DirectSolrSpellCheckerTest, HttpSolrCallGetCoreTest, NodeLostTriggerTest, SolrShardReporterTest, SolrCoreCheckLockOnStartupTest, TestExportWriter, SpellCheckCollatorWithCollapseTest, DistanceFunctionTest, TestTrie, TestConfigSets, BadComponentTest, TestRawTransformer, TestMaxScoreQueryParser, ChaosMonkeyNothingIsSafeTest, AutoAddReplicasIntegrationTest, TestSolrFieldCacheBean, TestLRUStatsCache, InfixSuggestersTest, TestTolerantUpdateProcessorRandomCloud, CdcrRequestHandlerTest, CopyFieldTest, DeleteNodeTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, OpenCloseCoreStressTest, ShardRoutingTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, TestRandomFaceting, LeaderElectionTest, TestDistributedSearch, TestFaceting, TermVectorComponentDistributedTest, ZkControllerTest, TestStressReorder, TestJoin, TestReload, TestStressVersions, SimpleFacetsTest, SpellCheckComponentTest, ConvertedLegacyTest, TestFiltering, TestIndexSearcher, ShowFileRequestHandlerTest, SimplePostToolTest, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SpellCheckCollatorTest, TestStressLucene, TestCSVLoader, SchemaVersionSpecificBehaviorTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, DebugComponentTest, TestSurroundQueryParser, TestWriterPerf, DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, PathHierarchyTokenizerFactoryTest, TermVectorComponentTest, TestIndexingPerformance, TestSolrQueryParser, ReturnFieldsTest, MBeansHandlerTest, HighlighterConfigTest, SOLR749Test, AlternateDirectoryTest, ResponseLogComponentTest, SolrIndexConfigTest, TestMergePolicyConfig, TestSolrDeletionPolicy2, TestDocSet, TestSearchPerf, TestSolrCoreProperties, TestPostingsSolrHighlighter, TestLuceneMatchVersion, SpellPossibilityIteratorTest, TestCharFilters, TestCodecSupport, SynonymTokenizerTest, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, TimeZoneUtilsTest, ResourceLoaderTest, PluginInfoTest, DateFieldTest, SpellingQueryConverterTest, RAMDirectoryFactoryTest, ClusterStateTest, TestLRUCache, ZkNodePropsTest, SliceStateTest, SystemInfoHandlerTest, UUIDFieldTest, DistributedMLTComponentTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestHighlightDedupGrouping, TestLuceneIndexBackCompat, AssignBackwardCompatibilityTest, CleanupOldIndexTest, CloudExitableDirectoryReaderTest, CollectionReloadTest, CollectionStateFormat2Test, CollectionTooManyReplicasTest, ConcurrentDeleteAndCreateCollectionTest, CreateCollectionCleanupTest, DeleteInactiveReplicaTest, DeleteLastCustomShardedReplicaTest, DeleteStatusTest, DistributedQueueTest, DocValuesNotIndexedTest, ForceLeaderTest, MigrateRouteKeyTest, NodeMutatorTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerStatusTest, OverseerTaskQueueTest, TestCloudDeleteByQuery, TestCloudInspectUtil, TestCloudPseudoReturnFields, TestConfigSetsAPI, TestLeaderElectionZkExpiry, TestOnReconnectListenerSupport, TestPullReplicaErrorHandling, TestStressInPlaceUpdates, TestPolicyCloud, CdcrReplicationDistributedZkTest, CdcrVersionReplicationTest, HdfsChaosMonkeyNothingIsSafeTest, HdfsRecoverLeaseTest, HdfsTlogReplayBufferedWhileIndexingTest, HdfsUnloadDistributedZkTest, HdfsWriteToMultipleCollectionsTest, ZkStateWriterTest, TestDynamicLoading, TestSQLHandler, TestConfigsApi, TestCoreAdminApis, DistributedDebugComponentTest, DistributedExpandComponentTest, DistributedFacetPivotLargeTest, DistributedFacetPivotSmallTest, ReplicaListTransformerTest, ShufflingReplicaListTransformerTest, SuggestComponentContextFilterQueryTest, TestPivotHelperCode, TestTrackingShardHandlerFactory, SolrGraphiteReporterTest, RegexBytesRefFilterTest, SubstringBytesRefFilterTest, TestStreamBody, TestMacros, TestBinaryResponseWriter, TestCustomDocTransformer, TestGeoJSONResponseWriter, TestGraphMLResponseWriter, TestJavabinTupleStreamParser, TestPushWriter, TestRetrieveFieldsOptimizer, TestSubQueryTransformerCrossCore, TestManagedResourceStorage, TestDynamicFieldResource, TestFieldResource, TestSchemaSimilarityResource, TestManagedSynonymFilterFactory, ManagedSchemaRoundRobinCloudTest, SpatialRPTFieldTypeTest, TestPointFields, TestComplexPhraseLeadingWildcard, TestFilteredDocIdSet, TestPayloadScoreQParserPlugin, TestReloadDeadlock, TestSmileRequest, TestXmlQParserPlugin, TestOrdValues, BlockJoinFacetDistribTest, BlockJoinFacetRandomTest, GraphQueryTest, TestScoreJoinQPNoScore, TestClassicSimilarityFactory, TestNonDefinedSimilarityFactory, TestDefaultStatsCache, TestExactSharedStatsCache, BasicAuthIntegrationTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicAuthIntegrationTest -Dtests.seed=2E4EF436C9211EC0 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ko-KR -Dtests.timezone=America/Jamaica -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | BasicAuthIntegrationTest (suite) <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:37501/solr: create the collection time out:180s
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2E4EF436C9211EC0]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
   [junit4]    > 	at org.apache.solr.security.BasicAuthIntegrationTest.setupCluster(BasicAuthIntegrationTest.java:82)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4] Completed [764/764 (4!)] on J2 in 214.90s, 0 tests, 1 error <<< FAILURES!

[...truncated 48094 lines...]

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk1.8.0_144) - Build # 1104 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1104/
Java: 64bit/jdk1.8.0_144 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.security.TestAuthorizationFramework.authorizationFrameworkTest

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:43539

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:43539
	at __randomizedtesting.SeedInfo.seed([CB22707201955A52:858105A1104E4B42]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:320)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 43 more




Build Log:
[...truncated 12269 lines...]
   [junit4] Suite: org.apache.solr.security.TestAuthorizationFramework
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.security.TestAuthorizationFramework_CB22707201955A52-001/init-core-data-001
   [junit4]   2> 13691 INFO  (SUITE-TestAuthorizationFramework-seed#[CB22707201955A52]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 13691 INFO  (SUITE-TestAuthorizationFramework-seed#[CB22707201955A52]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 13692 INFO  (SUITE-TestAuthorizationFramework-seed#[CB22707201955A52]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 13692 INFO  (SUITE-TestAuthorizationFramework-seed#[CB22707201955A52]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 13695 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 13695 INFO  (Thread-45) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 13695 INFO  (Thread-45) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 13697 ERROR (Thread-45) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 13795 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.ZkTestServer start zk server on port:41159
   [junit4]   2> 13800 INFO  (zkConnectionManagerCallback-34-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 13807 INFO  (zkConnectionManagerCallback-36-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 13817 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 13820 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 13824 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 13827 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 13830 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 13834 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 13836 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 13838 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 13840 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 13842 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 13844 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 13848 INFO  (zkConnectionManagerCallback-39-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 13851 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 13856 INFO  (zkConnectionManagerCallback-42-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 13958 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 13959 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@48b5d25c{/,null,AVAILABLE}
   [junit4]   2> 13961 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@9bb8a5d{SSL,[ssl, http/1.1]}{127.0.0.1:43539}
   [junit4]   2> 13961 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.e.j.s.Server Started @16451ms
   [junit4]   2> 13961 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.security.TestAuthorizationFramework_CB22707201955A52-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=43539, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.security.TestAuthorizationFramework_CB22707201955A52-001/control-001/cores}
   [junit4]   2> 13962 ERROR (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 13962 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 13962 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 13962 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 13962 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-02T19:07:27.727Z
   [junit4]   2> 13963 INFO  (zkConnectionManagerCallback-44-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 13964 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 13964 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.security.TestAuthorizationFramework_CB22707201955A52-001/control-001/solr.xml
   [junit4]   2> 13967 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 13967 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 13968 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@ea5072f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 13971 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41159/solr
   [junit4]   2> 13972 INFO  (zkConnectionManagerCallback-48-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 13975 INFO  (zkConnectionManagerCallback-50-thread-1-processing-n:127.0.0.1:43539_) [n:127.0.0.1:43539_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 14019 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [n:127.0.0.1:43539_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 14019 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [n:127.0.0.1:43539_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:43539_
   [junit4]   2> 14020 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [n:127.0.0.1:43539_    ] o.a.s.c.Overseer Overseer (id=99281800230404102-127.0.0.1:43539_-n_0000000000) starting
   [junit4]   2> 14028 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [n:127.0.0.1:43539_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43539_
   [junit4]   2> 14039 INFO  (zkCallback-49-thread-1-processing-n:127.0.0.1:43539_) [n:127.0.0.1:43539_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 14072 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [n:127.0.0.1:43539_    ] o.a.s.c.CoreContainer Initializing authorization plugin: org.apache.solr.security.MockAuthorizationPlugin
   [junit4]   2> 14099 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [n:127.0.0.1:43539_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ea5072f
   [junit4]   2> 14108 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [n:127.0.0.1:43539_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ea5072f
   [junit4]   2> 14108 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [n:127.0.0.1:43539_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ea5072f
   [junit4]   2> 14110 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [n:127.0.0.1:43539_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.security.TestAuthorizationFramework_CB22707201955A52-001/control-001/cores
   [junit4]   2> 14132 INFO  (zkConnectionManagerCallback-56-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 14133 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 14133 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41159/solr ready
   [junit4]   2> 14151 INFO  (qtp1027968142-194) [n:127.0.0.1:43539_    ] o.a.s.s.MockAuthorizationPlugin User request: null
   [junit4]   2> 14152 INFO  (qtp1027968142-194) [n:127.0.0.1:43539_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:43539_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 14160 INFO  (OverseerThreadFactory-96-thread-1-processing-n:127.0.0.1:43539_) [n:127.0.0.1:43539_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 14289 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_    ] o.a.s.s.MockAuthorizationPlugin User request: null
   [junit4]   2> 14291 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 14292 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 14404 INFO  (zkCallback-49-thread-1-processing-n:127.0.0.1:43539_) [n:127.0.0.1:43539_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 15335 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 15370 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 15375 WARN  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.TrieIntField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 15378 WARN  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.TrieFloatField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 15381 WARN  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.TrieLongField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 15389 WARN  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.TrieDateField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 15463 WARN  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.EnumField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 15473 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 15487 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 15487 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ea5072f
   [junit4]   2> 15487 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 15488 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.security.TestAuthorizationFramework_CB22707201955A52-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.security.TestAuthorizationFramework_CB22707201955A52-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 15491 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=30, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 15502 WARN  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 15535 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 15535 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 15536 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 15536 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 15537 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=12, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=60.234375, floorSegmentMB=0.9609375, forceMergeDeletesPctAllowed=6.187933609446873, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 15537 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7126ba7d[control_collection_shard1_replica_n1] main]
   [junit4]   2> 15539 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 15539 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 15540 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 15542 INFO  (searcherExecutor-99-thread-1-processing-n:127.0.0.1:43539_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@7126ba7d[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 15543 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588508805623185408
   [junit4]   2> 15551 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 15551 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 15551 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:43539/control_collection_shard1_replica_n1/
   [junit4]   2> 15552 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 15552 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:43539/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 15552 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 15560 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:43539/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 74208 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.ChaosMonkey monkey: stop jetty! 43539
   [junit4]   2> 74208 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=376227798
   [junit4]   2> 74208 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 74208 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@58b141c4: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@2e35d7f7
   [junit4]   2> 74216 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 74216 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7a9dcc74: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@32a99420
   [junit4]   2> 74222 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 74222 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@356c1bb0: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@2b9fe0da
   [junit4]   2> 74225 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:43539_
   [junit4]   2> 74226 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 74227 INFO  (zkCallback-49-thread-2-processing-n:127.0.0.1:43539_) [n:127.0.0.1:43539_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 74228 INFO  (coreCloseExecutor-104-thread-1) [n:127.0.0.1:43539_ c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1b093f84
   [junit4]   2> 74228 INFO  (coreCloseExecutor-104-thread-1) [n:127.0.0.1:43539_ c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.control_collection.shard1.replica_n1, tag=453590916
   [junit4]   2> 74228 INFO  (coreCloseExecutor-104-thread-1) [n:127.0.0.1:43539_ c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1307fda4: rootName = null, domain = solr.core.control_collection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.control_collection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@7a8271d2
   [junit4]   2> 74242 INFO  (coreCloseExecutor-104-thread-1) [n:127.0.0.1:43539_ c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.control_collection.shard1.leader, tag=453590916
   [junit4]   2> 85065 INFO  (ScheduledTrigger-94-thread-1) [n:127.0.0.1:43539_    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 85084 INFO  (AutoscalingActionExecutor-95-thread-1-processing-n:127.0.0.1:43539_) [n:127.0.0.1:43539_    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 2621260822 sessionWrapper.createTime 2621260815978169, this.sessionWrapper.createTime 2621260815978169 
   [junit4]   2> 85085 INFO  (AutoscalingActionExecutor-95-thread-1-processing-n:127.0.0.1:43539_) [n:127.0.0.1:43539_    ] o.a.s.c.a.ExecutePlanAction No operations to execute for event: {
   [junit4]   2>   "id":"9500382732e18Tar2oeg77c1j1mlu255s3cjixa",
   [junit4]   2>   "source":".auto_add_replicas",
   [junit4]   2>   "eventTime":2621250794106392,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2621250794106392],
   [junit4]   2>     "_enqueue_time_":2621260798529189,
   [junit4]   2>     "nodeNames":["127.0.0.1:43539_"]}}
   [junit4]   2> 194158 ERROR (qtp1027968142-194) [n:127.0.0.1:43539_    ] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: create the collection time out:180s
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:314)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:246)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:224)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 194159 INFO  (qtp1027968142-194) [n:127.0.0.1:43539_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:43539_&wt=javabin&version=2} status=500 QTime=180007
   [junit4]   2> 194159 ERROR (qtp1027968142-194) [n:127.0.0.1:43539_    ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: create the collection time out:180s
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:314)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:246)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:224)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 614387 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.Overseer Overseer (id=99281800230404102-127.0.0.1:43539_-n_0000000000) closing
   [junit4]   2> 614387 INFO  (OverseerStateUpdate-99281800230404102-127.0.0.1:43539_-n_0000000000) [n:127.0.0.1:43539_    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:43539_
   [junit4]   2> 614388 ERROR (OverseerThreadFactory-96-thread-1-processing-n:127.0.0.1:43539_) [n:127.0.0.1:43539_    ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: https://127.0.0.1:43539
   [junit4]   2> org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:43539
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]   2> 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]   2> 	at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:172)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]   2> 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]   2> 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]   2> 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]   2> 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]   2> 	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
   [junit4]   2> 	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
   [junit4]   2> 	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
   [junit4]   2> 	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
   [junit4]   2> 	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
   [junit4]   2> 	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   [junit4]   2> 	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   [junit4]   2> 	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]   2> 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]   2> 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]   2> 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]   2> 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:118)
   [junit4]   2> 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]   2> 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]   2> 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]   2> 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
   [junit4]   2> 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]   2> 	... 12 more
   [junit4]   2> 614391 ERROR (OverseerThreadFactory-96-thread-1-processing-n:127.0.0.1:43539_) [n:127.0.0.1:43539_    ] o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [control_collection].
   [junit4]   2> 614395 INFO  (OverseerThreadFactory-96-thread-1-processing-n:127.0.0.1:43539_) [n:127.0.0.1:43539_    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 644494 INFO  (zkCallback-49-thread-5-processing-n:127.0.0.1:43539_) [n:127.0.0.1:43539_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 644496 ERROR (OverseerThreadFactory-96-thread-1-processing-n:127.0.0.1:43539_) [n:127.0.0.1:43539_    ] o.a.s.c.OverseerCollectionMessageHandler Collection: control_collection operation: create failed:org.apache.solr.common.SolrException: Could not fully remove collection: control_collection
   [junit4]   2> 	at org.apache.solr.cloud.DeleteCollectionCmd.call(DeleteCollectionCmd.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionMessageHandler.cleanupCollection(OverseerCollectionMessageHandler.java:688)
   [junit4]   2> 	at org.apache.solr.cloud.CreateCollectionCmd.call(CreateCollectionCmd.java:240)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:242)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:464)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 644525 INFO  (OverseerThreadFactory-96-thread-1-processing-n:127.0.0.1:43539_) [n:127.0.0.1:43539_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 644526 WARN  (OverseerAutoScalingTriggerThread-99281800230404102-127.0.0.1:43539_-n_0000000000) [n:127.0.0.1:43539_    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 644531 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@9bb8a5d{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 644531 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@48b5d25c{/,null,UNAVAILABLE}
   [junit4]   2> 644566 ERROR (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController Error getting leader from zk
   [junit4]   2> org.apache.solr.common.SolrException: No registered leader was found after waiting for 1220000ms , collection: control_collection slice: shard1 saw state=null with live_nodes=[]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:771)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:736)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1178)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1079)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1014)
   [junit4]   2> 	at org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:181)
   [junit4]   2> 	at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:208)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:888)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1049)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:950)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 644567 ERROR (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkContainer :org.apache.solr.common.SolrException: Error getting leader from zk for shard shard1
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1207)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1079)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1014)
   [junit4]   2> 	at org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:181)
   [junit4]   2> 	at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:208)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:888)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1049)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:950)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: No registered leader was found after waiting for 1220000ms , collection: control_collection slice: shard1 saw state=null with live_nodes=[]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:771)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:736)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1178)
   [junit4]   2> 	... 44 more
   [junit4]   2> 
   [junit4]   2> 644579 INFO  (qtp1027968142-189) [n:127.0.0.1:43539_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=630290
   [junit4]   2> 644579 ERROR (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 644579 INFO  (TEST-TestAuthorizationFramework.authorizationFrameworkTest-seed#[CB22707201955A52]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41159 41159
   [junit4]   2> 644979 INFO  (Thread-45) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41159 41159
   [junit4]   2> 644979 WARN  (Thread-45) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/aliases.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/clusterprops.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/live_nodes
   [junit4]   2> 	2	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestAuthorizationFramework -Dtests.method=authorizationFrameworkTest -Dtests.seed=CB22707201955A52 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=pt -Dtests.timezone=America/Sitka -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR    631s J0 | TestAuthorizationFramework.authorizationFrameworkTest <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:43539
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CB22707201955A52:858105A1104E4B42]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:320)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]    > 	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
   [junit4]    > 	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
   [junit4]    > 	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
   [junit4]    > 	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
   [junit4]    > 	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]    > 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]    > 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]    > 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]    > 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]    > 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]    > 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
   [junit4]    > 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]    > 	... 43 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.security.TestAuthorizationFramework_CB22707201955A52-001
   [junit4]   2> Jan 02, 2018 7:17:58 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=8, maxDocsPerChunk=4, blockSize=206), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=8, blockSize=206)), sim=RandomSimilarity(queryNorm=true): {}, locale=pt, timezone=America/Sitka
   [junit4]   2> NOTE: Linux 4.10.0-40-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=8,threads=1,free=335171808,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestXmlQParser, TestClusterProperties, TestSolrJ, TestManagedResource, TestIBSimilarityFactory, TestConfigSetImmutable, TestPushWriter, BasicFunctionalityTest, TestReRankQParserPlugin, TestSha256AuthenticationProvider, TestAuthorizationFramework]
   [junit4] Completed [151/764 (1!)] on J0 in 631.57s, 1 test, 1 error <<< FAILURES!

[...truncated 50106 lines...]

[JENKINS] Lucene-Solr-7.x-Linux (32bit/jdk1.8.0_144) - Build # 1103 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1103/
Java: 32bit/jdk1.8.0_144 -server -XX:+UseG1GC

2 tests failed.
FAILED:  org.apache.solr.cloud.TestShortCircuitedRequests.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:38249

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:38249
	at __randomizedtesting.SeedInfo.seed([3C248CD33396D812:B470B3099D6AB5EA]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:320)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 43 more


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager.testGetSetRemoveData

Error Message:
Node watch should have fired!

Stack Trace:
java.lang.AssertionError: Node watch should have fired!
	at __randomizedtesting.SeedInfo.seed([3C248CD33396D812:1AB4638167181E18]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager.testGetSetRemoveData(TestDistribStateManager.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13533 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager
   [junit4]   2> 2743161 INFO  (SUITE-TestDistribStateManager-seed#[3C248CD33396D812]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestDistribStateManager_3C248CD33396D812-001/init-core-data-001
   [junit4]   2> 2743161 WARN  (SUITE-TestDistribStateManager-seed#[3C248CD33396D812]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=31 numCloses=31
   [junit4]   2> 2743161 INFO  (SUITE-TestDistribStateManager-seed#[3C248CD33396D812]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2743163 INFO  (SUITE-TestDistribStateManager-seed#[3C248CD33396D812]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2743165 INFO  (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[3C248CD33396D812]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testGetSetRemoveData
   [junit4]   2> 2743165 INFO  (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2743165 INFO  (Thread-6700) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2743165 INFO  (Thread-6700) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2743166 ERROR (Thread-6700) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2743265 INFO  (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer start zk server on port:40069
   [junit4]   2> 2743267 INFO  (zkConnectionManagerCallback-9800-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2743267 INFO  (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[3C248CD33396D812]) [    ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.client.solrj.impl.ZkDistribStateManager
   [junit4]   2> 2743270 INFO  (zkConnectionManagerCallback-9802-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2743271 INFO  (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[3C248CD33396D812]) [    ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.client.solrj.impl.ZkDistribStateManager
   [junit4]   2> 2748271 ERROR (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[3C248CD33396D812]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2748272 INFO  (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40069 40069
   [junit4]   2> 2748303 INFO  (Thread-6700) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40069 40069
   [junit4]   2> 2748304 INFO  (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[3C248CD33396D812]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testGetSetRemoveData
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDistribStateManager -Dtests.method=testGetSetRemoveData -Dtests.seed=3C248CD33396D812 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it-IT -Dtests.timezone=Etc/GMT+12 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 5.14s J2 | TestDistribStateManager.testGetSetRemoveData <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Node watch should have fired!
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3C248CD33396D812:1AB4638167181E18]:0)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager.testGetSetRemoveData(TestDistribStateManager.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2748307 INFO  (TEST-TestDistribStateManager.testCreateMode-seed#[3C248CD33396D812]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreateMode
   [junit4]   2> 2748307 INFO  (TEST-TestDistribStateManager.testCreateMode-seed#[3C248CD33396D812]) [    ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.cloud.autoscaling.sim.SimDistribStateManager
   [junit4]   2> 2748308 INFO  (TEST-TestDistribStateManager.testCreateMode-seed#[3C248CD33396D812]) [    ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.cloud.autoscaling.sim.SimDistribStateManager
   [junit4]   2> 2748308 INFO  (TEST-TestDistribStateManager.testCreateMode-seed#[3C248CD33396D812]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreateMode
   [junit4]   2> 2748310 INFO  (TEST-TestDistribStateManager.testHasData-seed#[3C248CD33396D812]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testHasData
   [junit4]   2> 2748310 INFO  (TEST-TestDistribStateManager.testHasData-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2748311 INFO  (Thread-6701) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2748311 INFO  (Thread-6701) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2748313 ERROR (Thread-6701) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2748411 INFO  (TEST-TestDistribStateManager.testHasData-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer start zk server on port:37203
   [junit4]   2> 2748412 INFO  (zkConnectionManagerCallback-9804-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748412 INFO  (TEST-TestDistribStateManager.testHasData-seed#[3C248CD33396D812]) [    ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.client.solrj.impl.ZkDistribStateManager
   [junit4]   2> 2748414 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x160b7d5f0c60000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2748414 ERROR (TEST-TestDistribStateManager.testHasData-seed#[3C248CD33396D812]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2748415 INFO  (TEST-TestDistribStateManager.testHasData-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37203 37203
   [junit4]   2> 2758456 INFO  (Thread-6701) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37203 37203
   [junit4]   2> 2758456 INFO  (TEST-TestDistribStateManager.testHasData-seed#[3C248CD33396D812]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testHasData
   [junit4]   2> 2758459 INFO  (TEST-TestDistribStateManager.testListData-seed#[3C248CD33396D812]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testListData
   [junit4]   2> 2758459 INFO  (TEST-TestDistribStateManager.testListData-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2758459 INFO  (Thread-6702) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2758459 INFO  (Thread-6702) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2758464 ERROR (Thread-6702) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2758559 INFO  (TEST-TestDistribStateManager.testListData-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer start zk server on port:35077
   [junit4]   2> 2758561 INFO  (zkConnectionManagerCallback-9806-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2758561 INFO  (TEST-TestDistribStateManager.testListData-seed#[3C248CD33396D812]) [    ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.client.solrj.impl.ZkDistribStateManager
   [junit4]   2> 2758564 ERROR (TEST-TestDistribStateManager.testListData-seed#[3C248CD33396D812]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2758564 INFO  (TEST-TestDistribStateManager.testListData-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35077 35077
   [junit4]   2> 2763851 INFO  (Thread-6702) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35077 35077
   [junit4]   2> 2763852 INFO  (TEST-TestDistribStateManager.testListData-seed#[3C248CD33396D812]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testListData
   [junit4]   2> 2763854 INFO  (TEST-TestDistribStateManager.testMulti-seed#[3C248CD33396D812]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMulti
   [junit4]   2> 2763854 INFO  (TEST-TestDistribStateManager.testMulti-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2763854 INFO  (Thread-6703) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2763854 INFO  (Thread-6703) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2763855 ERROR (Thread-6703) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2763954 INFO  (TEST-TestDistribStateManager.testMulti-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer start zk server on port:43713
   [junit4]   2> 2763956 INFO  (zkConnectionManagerCallback-9808-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2763956 INFO  (TEST-TestDistribStateManager.testMulti-seed#[3C248CD33396D812]) [    ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.client.solrj.impl.ZkDistribStateManager
   [junit4]   2> 2763956 ERROR (TEST-TestDistribStateManager.testMulti-seed#[3C248CD33396D812]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2763957 INFO  (TEST-TestDistribStateManager.testMulti-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43713 43713
   [junit4]   2> 2769094 INFO  (Thread-6703) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43713 43713
   [junit4]   2> 2769095 INFO  (TEST-TestDistribStateManager.testMulti-seed#[3C248CD33396D812]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testMulti
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestDistribStateManager_3C248CD33396D812-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1590, maxMBSortInHeap=5.916951425229057, sim=RandomSimilarity(queryNorm=true): {}, locale=it-IT, timezone=Etc/GMT+12
   [junit4]   2> NOTE: Linux 4.10.0-40-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=1,free=228717872,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestSchemaVersionResource, IndexBasedSpellCheckerTest, TestNumericRangeQuery64, TestJsonRequest, TestRandomRequestDistribution, TestGenericDistributedQueue, EchoParamsTest, HLLSerializationTest, HdfsDirectoryFactoryTest, AnalyticsMergeStrategyTest, TestSolrXml, LoggingHandlerTest, TestLMJelinekMercerSimilarityFactory, TestLFUCache, TestQuerySenderListener, TestRawTransformer, SimplePostToolTest, TestDistribDocBasedVersion, HdfsRecoveryZkTest, SolrCoreCheckLockOnStartupTest, TestFiltering, TestCloudDeleteByQuery, SmileWriterTest, ScriptEngineTest, ZkStateReaderTest, TestSolrDeletionPolicy2, SuggesterTest, TestMinMaxOnMultiValuedField, AlternateDirectoryTest, SegmentsInfoRequestHandlerTest, TestStressLiveNodes, DirectSolrConnectionTest, CoreAdminHandlerTest, RecoveryZkTest, TestMacroExpander, CurrencyFieldTypeTest, TestSolrConfigHandlerConcurrent, RuleEngineTest, TestPerFieldSimilarityWithDefaultOverride, BlockCacheTest, AtomicUpdateProcessorFactoryTest, TriLevelCompositeIdRoutingTest, NodeMutatorTest, TestSolrFieldCacheBean, TestDistributedGrouping, TestDistributedStatsComponentCardinality, TestPivotHelperCode, UniqFieldsUpdateProcessorFactoryTest, TestHdfsUpdateLog, UUIDFieldTest, TestSearchPerf, TestConfigSets, BasicDistributedZk2Test, TestMaxTokenLenTokenizer, CollectionsAPIAsyncDistributedZkTest, LargeFieldTest, TestSQLHandler, TestCloudPivotFacet, AsyncCallRequestStatusResponseTest, DistributedFacetPivotLargeTest, ConfigSetsAPITest, ChaosMonkeyNothingIsSafeTest, ClusterStateTest, SolrCloudExampleTest, TestHashPartitioner, EnumFieldTest, TestClassicSimilarityFactory, TestHashQParserPlugin, SolrMetricManagerTest, TestFuzzyAnalyzedSuggestions, LeaderInitiatedRecoveryOnCommitTest, HdfsThreadLeakTest, FieldAnalysisRequestHandlerTest, MigrateRouteKeyTest, SolrShardReporterTest, FullSolrCloudDistribCmdsTest, TestCloudSearcherWarming, TestSolrCLIRunExample, HdfsBasicDistributedZkTest, TestFieldResource, CdcrUpdateLogTest, TestTrieFacet, TestSmileRequest, TestNestedDocsSort, TestLRUCache, DistributedFacetPivotSmallTest, SharedFSAutoReplicaFailoverTest, CdcrRequestHandlerTest, TestNoOpRegenerator, TestFieldSortValues, AliasIntegrationTest, TestEmbeddedSolrServerConstructors, RequiredFieldsTest, TestComputePlanAction, IndexSchemaTest, DirectSolrSpellCheckerTest, TestSolrCoreProperties, TestTolerantUpdateProcessorRandomCloud, TestSchemalessBufferedUpdates, DistributedFacetSimpleRefinementLongTailTest, HighlighterMaxOffsetTest, FieldMutatingUpdateProcessorTest, HighlighterTest, TestConfigOverlay, DeleteNodeTest, HdfsDirectoryTest, TestReplicationHandlerBackup, SolrJmxReporterTest, BitVectorTest, TestMaxScoreQueryParser, CoreAdminRequestStatusTest, SolrGangliaReporterTest, TestRestManager, TestSystemIdResolver, RecoveryAfterSoftCommitTest, HttpSolrCallGetCoreTest, TestChildDocTransformer, PluginInfoTest, QueryElevationComponentTest, ConnectionReuseTest, CSVRequestHandlerTest, HdfsWriteToMultipleCollectionsTest, TestSolrCloudWithSecureImpersonation, ZkNodePropsTest, UnloadDistributedZkTest, OpenCloseCoreStressTest, OverseerTest, LeaderElectionTest, TestZkChroot, TestRandomDVFaceting, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, TestReload, TestRangeQuery, TestGroupingSearch, TestFunctionQuery, DirectUpdateHandlerTest, TestIndexSearcher, SoftAutoCommitTest, TestCodecSupport, TestLMDirichletSimilarityFactory, TimeZoneUtilsTest, TestFastLRUCache, PreAnalyzedFieldTest, SliceStateTest, DistributedMLTComponentTest, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, TestEmbeddedSolrServerAdminHandler, TestEmbeddedSolrServerSchemaAPI, ChaosMonkeySafeLeaderWithPullReplicasTest, CollectionsAPISolrJTest, DeleteStatusTest, DocValuesNotIndexedTest, LeaderElectionContextKeyTest, LeaderFailoverAfterPartitionTest, LeaderInitiatedRecoveryOnShardRestartTest, MissingSegmentRecoveryTest, MoveReplicaHDFSFailoverTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerRolesTest, PeerSyncReplicationTest, ShardSplitTest, TestPullReplicaErrorHandling, TestRequestForwarding, TestSSLRandomization, TestSegmentSorting, TestSolrCloudWithKerberosAlt, TestStressInPlaceUpdates, AutoAddReplicasIntegrationTest, HttpTriggerListenerTest, MetricTriggerTest, NodeLostTriggerTest, TestPolicyCloud, TestDistribStateManager]
   [junit4] Completed [527/764 (1!)] on J2 in 25.95s, 5 tests, 1 failure <<< FAILURES!

[...truncated 5 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestShortCircuitedRequests
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestShortCircuitedRequests_3C248CD33396D812-001/init-core-data-001
   [junit4]   2> 2167677 WARN  (SUITE-TestShortCircuitedRequests-seed#[3C248CD33396D812]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=9 numCloses=9
   [junit4]   2> 2167677 INFO  (SUITE-TestShortCircuitedRequests-seed#[3C248CD33396D812]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2167678 INFO  (SUITE-TestShortCircuitedRequests-seed#[3C248CD33396D812]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2167678 INFO  (SUITE-TestShortCircuitedRequests-seed#[3C248CD33396D812]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2167678 INFO  (SUITE-TestShortCircuitedRequests-seed#[3C248CD33396D812]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2167680 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2167680 INFO  (Thread-1539) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2167680 INFO  (Thread-1539) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2167681 ERROR (Thread-1539) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2167780 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer start zk server on port:34211
   [junit4]   2> 2167782 INFO  (zkConnectionManagerCallback-1149-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2167784 INFO  (zkConnectionManagerCallback-1151-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2167787 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2167788 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 2167788 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2167789 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2167789 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2167790 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2167790 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2167791 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2167791 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2167792 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2167793 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2167797 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x160b7cd1a4c0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2167797 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 2167883 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2167884 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@b1361e{/,null,AVAILABLE}
   [junit4]   2> 2167885 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@21f393{HTTP/1.1,[http/1.1]}{127.0.0.1:38249}
   [junit4]   2> 2167885 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.e.j.s.Server Started @2170707ms
   [junit4]   2> 2167885 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestShortCircuitedRequests_3C248CD33396D812-001/tempDir-001/control/data, hostContext=/, hostPort=38249, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestShortCircuitedRequests_3C248CD33396D812-001/control-001/cores}
   [junit4]   2> 2167885 ERROR (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2167885 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 2167885 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2167885 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2167885 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-02T16:56:00.536Z
   [junit4]   2> 2167887 INFO  (zkConnectionManagerCallback-1153-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2167887 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2167888 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestShortCircuitedRequests_3C248CD33396D812-001/control-001/solr.xml
   [junit4]   2> 2167890 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2167890 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2167891 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@13f131, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2167893 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34211/solr
   [junit4]   2> 2167894 INFO  (zkConnectionManagerCallback-1157-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2167897 INFO  (zkConnectionManagerCallback-1159-thread-1-processing-n:127.0.0.1:38249_) [n:127.0.0.1:38249_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2167924 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [n:127.0.0.1:38249_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2167925 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [n:127.0.0.1:38249_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38249_
   [junit4]   2> 2167925 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [n:127.0.0.1:38249_    ] o.a.s.c.Overseer Overseer (id=99281283339517956-127.0.0.1:38249_-n_0000000000) starting
   [junit4]   2> 2167929 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [n:127.0.0.1:38249_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38249_
   [junit4]   2> 2167933 INFO  (zkCallback-1158-thread-1-processing-n:127.0.0.1:38249_) [n:127.0.0.1:38249_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2167985 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [n:127.0.0.1:38249_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13f131
   [junit4]   2> 2167992 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [n:127.0.0.1:38249_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13f131
   [junit4]   2> 2167992 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [n:127.0.0.1:38249_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13f131
   [junit4]   2> 2167993 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [n:127.0.0.1:38249_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestShortCircuitedRequests_3C248CD33396D812-001/control-001/cores
   [junit4]   2> 2168009 INFO  (zkConnectionManagerCallback-1165-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2168010 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2168010 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34211/solr ready
   [junit4]   2> 2168011 INFO  (qtp4178871-12236) [n:127.0.0.1:38249_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:38249_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2168012 INFO  (OverseerThreadFactory-2798-thread-1-processing-n:127.0.0.1:38249_) [n:127.0.0.1:38249_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2168115 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2168116 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2168221 INFO  (zkCallback-1158-thread-1-processing-n:127.0.0.1:38249_) [n:127.0.0.1:38249_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2169135 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 2169142 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2169189 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 2169196 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2169196 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13f131
   [junit4]   2> 2169196 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2169196 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestShortCircuitedRequests_3C248CD33396D812-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestShortCircuitedRequests_3C248CD33396D812-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2169198 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=2, maxMergedSegmentMB=7.921875, floorSegmentMB=0.720703125, forceMergeDeletesPctAllowed=2.2861842216645845, segmentsPerTier=44.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2169200 WARN  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2169228 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2169228 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2169229 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2169229 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2169230 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8349892895685721]
   [junit4]   2> 2169230 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1b07202[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2169231 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2169231 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2169232 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2169233 INFO  (searcherExecutor-2801-thread-1-processing-n:127.0.0.1:38249_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@1b07202[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2169233 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588500535059677184
   [junit4]   2> 2169236 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2169236 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2169236 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:38249/control_collection_shard1_replica_n1/
   [junit4]   2> 2169236 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2169236 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:38249/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2169236 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2169237 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:38249/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2228062 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.ChaosMonkey monkey: stop jetty! 38249
   [junit4]   2> 2228062 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=29010118
   [junit4]   2> 2228062 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2228062 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1bc5b8c: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@455d0f
   [junit4]   2> 2228066 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2228066 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2b0aba: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@178f195
   [junit4]   2> 2228069 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2228069 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1f8561: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@67ce4f
   [junit4]   2> 2228070 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:38249_
   [junit4]   2> 2228070 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 2228071 INFO  (zkCallback-1158-thread-2-processing-n:127.0.0.1:38249_) [n:127.0.0.1:38249_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 2228071 INFO  (coreCloseExecutor-2806-thread-1) [n:127.0.0.1:38249_ c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@16a7fb1
   [junit4]   2> 2228071 INFO  (coreCloseExecutor-2806-thread-1) [n:127.0.0.1:38249_ c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.control_collection.shard1.replica_n1, tag=23756721
   [junit4]   2> 2228071 INFO  (coreCloseExecutor-2806-thread-1) [n:127.0.0.1:38249_ c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1e9469a: rootName = null, domain = solr.core.control_collection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.control_collection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@50a9c5
   [junit4]   2> 2228077 INFO  (coreCloseExecutor-2806-thread-1) [n:127.0.0.1:38249_ c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.control_collection.shard1.leader, tag=23756721
   [junit4]   2> 2238944 INFO  (ScheduledTrigger-2796-thread-1) [n:127.0.0.1:38249_    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 2238945 INFO  (AutoscalingActionExecutor-2797-thread-1-processing-n:127.0.0.1:38249_) [n:127.0.0.1:38249_    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 2613373568 sessionWrapper.createTime 2613373568745745, this.sessionWrapper.createTime 2613373568745745 
   [junit4]   2> 2238945 INFO  (AutoscalingActionExecutor-2797-thread-1-processing-n:127.0.0.1:38249_) [n:127.0.0.1:38249_    ] o.a.s.c.a.ExecutePlanAction No operations to execute for event: {
   [junit4]   2>   "id":"948d71eff07ffT2xiua3hsvd0k5hlcnaaktr8ds",
   [junit4]   2>   "source":".auto_add_replicas",
   [junit4]   2>   "eventTime":2613363565594623,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[2613363565594623],
   [junit4]   2>     "_enqueue_time_":2613373567506005,
   [junit4]   2>     "nodeNames":["127.0.0.1:38249_"]}}
   [junit4]   2> 2348013 ERROR (qtp4178871-12236) [n:127.0.0.1:38249_    ] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: create the collection time out:180s
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:314)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:246)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:224)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 2348014 INFO  (qtp4178871-12236) [n:127.0.0.1:38249_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:38249_&wt=javabin&version=2} status=500 QTime=180002
   [junit4]   2> 2348014 ERROR (qtp4178871-12236) [n:127.0.0.1:38249_    ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: create the collection time out:180s
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:314)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:246)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:224)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 2768213 ERROR (OverseerThreadFactory-2798-thread-1-processing-n:127.0.0.1:38249_) [n:127.0.0.1:38249_    ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: http://127.0.0.1:38249
   [junit4]   2> org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:38249
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]   2> 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]   2> 	at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:172)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]   2> 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]   2> 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]   2> 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]   2> 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]   2> 	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   [junit4]   2> 	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   [junit4]   2> 	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]   2> 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]   2> 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]   2> 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]   2> 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:118)
   [junit4]   2> 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]   2> 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]   2> 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]   2> 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
   [junit4]   2> 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]   2> 	... 12 more
   [junit4]   2> 2768213 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.Overseer Overseer (id=99281283339517956-127.0.0.1:38249_-n_0000000000) closing
   [junit4]   2> 2768214 INFO  (OverseerStateUpdate-99281283339517956-127.0.0.1:38249_-n_0000000000) [n:127.0.0.1:38249_    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:38249_
   [junit4]   2> 2768214 ERROR (OverseerThreadFactory-2798-thread-1-processing-n:127.0.0.1:38249_) [n:127.0.0.1:38249_    ] o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [control_collection].
   [junit4]   2> 2768214 INFO  (OverseerThreadFactory-2798-thread-1-processing-n:127.0.0.1:38249_) [n:127.0.0.1:38249_    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 2798250 ERROR (OverseerThreadFactory-2798-thread-1-processing-n:127.0.0.1:38249_) [n:127.0.0.1:38249_    ] o.a.s.c.OverseerCollectionMessageHandler Collection: control_collection operation: create failed:org.apache.solr.common.SolrException: Could not fully remove collection: control_collection
   [junit4]   2> 	at org.apache.solr.cloud.DeleteCollectionCmd.call(DeleteCollectionCmd.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionMessageHandler.cleanupCollection(OverseerCollectionMessageHandler.java:688)
   [junit4]   2> 	at org.apache.solr.cloud.CreateCollectionCmd.call(CreateCollectionCmd.java:240)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:242)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:464)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 2798253 INFO  (zkCallback-1158-thread-5-processing-n:127.0.0.1:38249_) [n:127.0.0.1:38249_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 2798259 INFO  (OverseerThreadFactory-2798-thread-1-processing-n:127.0.0.1:38249_) [n:127.0.0.1:38249_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2798261 WARN  (OverseerAutoScalingTriggerThread-99281283339517956-127.0.0.1:38249_-n_0000000000) [n:127.0.0.1:38249_    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 2798288 ERROR (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController Error getting leader from zk
   [junit4]   2> org.apache.solr.common.SolrException: No registered leader was found after waiting for 1220000ms , collection: control_collection slice: shard1 saw state=null with live_nodes=[]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:771)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:736)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1178)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1079)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1014)
   [junit4]   2> 	at org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:181)
   [junit4]   2> 	at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:208)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:888)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1049)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:950)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2798289 ERROR (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkContainer :org.apache.solr.common.SolrException: Error getting leader from zk for shard shard1
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1207)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1079)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.register(ZkController.java:1014)
   [junit4]   2> 	at org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:181)
   [junit4]   2> 	at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:208)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:888)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1049)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:950)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: No registered leader was found after waiting for 1220000ms , collection: control_collection slice: shard1 saw state=null with live_nodes=[]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:771)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:736)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1178)
   [junit4]   2> 	... 41 more
   [junit4]   2> 
   [junit4]   2> 2798293 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@21f393{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2798293 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@b1361e{/,null,UNAVAILABLE}
   [junit4]   2> 2798297 INFO  (qtp4178871-12238) [n:127.0.0.1:38249_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=630182
   [junit4]   2> 2798309 ERROR (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2798309 INFO  (TEST-TestShortCircuitedRequests.test-seed#[3C248CD33396D812]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34211 34211
   [junit4]   2> 2803525 INFO  (Thread-1539) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34211 34211
   [junit4]   2> 2803526 WARN  (Thread-1539) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/aliases.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/clusterprops.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/live_nodes
   [junit4]   2> 	2	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestShortCircuitedRequests -Dtests.method=test -Dtests.seed=3C248CD33396D812 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr-Latn-ME -Dtests.timezone=America/Santiago -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR    636s J0 | TestShortCircuitedRequests.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:38249
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3C248CD33396D812:B470B3099D6AB5EA]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:320)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   [junit4]    > 	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]    > 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]    > 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]    > 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]    > 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]    > 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]    > 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
   [junit4]    > 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]    > 	... 43 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestShortCircuitedRequests_3C248CD33396D812-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=326, maxMBSortInHeap=5.164369999877016, sim=RandomSimilarity(queryNorm=true): {}, locale=sr-Latn-ME, timezone=America/Santiago
   [junit4]   2> NOTE: Linux 4.10.0-40-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=1,free=271781944,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [BigEndianAscendingWordDeserializerTest, TestBadConfig, TestRequestStatusCollectionAPI, PeerSyncTest, BasicFunctionalityTest, QueryParsingTest, SolrMetricReporterTest, TestSolrJ, TestRandomFaceting, RegexBytesRefFilterTest, DistributedFacetPivotLongTailTest, TestIntervalFaceting, StressHdfsTest, CursorPagingTest, JsonLoaderTest, TestFastOutputStream, TestCloudSchemaless, TestFieldCacheReopen, HLLUtilTest, MetricsConfigTest, TestPhraseSuggestions, TestPHPSerializedResponseWriter, TestNumericTerms32, ExplicitHLLTest, CloudMLTQParserTest, TestInitQParser, DirectoryFactoryTest, TestOnReconnectListenerSupport, OverseerTaskQueueTest, TestQueryUtils, CachingDirectoryFactoryTest, TestExactSharedStatsCache, TestInPlaceUpdatesDistrib, TestOrdValues, ExternalFileFieldSortTest, TestExpandComponent, SpatialFilterTest, TestWordDelimiterFilterFactory, DistributedQueryElevationComponentTest, FastVectorHighlighterTest, TestFreeTextSuggestions, DataDrivenBlockJoinTest, SimpleCollectionCreateDeleteTest, ExecutePlanActionTest, TermsComponentTest, UpdateParamsTest, DeleteReplicaTest, TestAddFieldRealTimeGet, PrimitiveFieldTypeTest, StatelessScriptUpdateProcessorFactoryTest, SpellCheckComponentTest, SyncSliceTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestLeaderElectionWithEmptyReplica, AssignBackwardCompatibilityTest, DistribCursorPagingTest, ClassificationUpdateProcessorTest, TestClusterProperties, BasicDistributedZkTest, TestCoreBackup, TestGeoJSONResponseWriter, IndexSchemaRuntimeFieldTest, TestSolrCloudWithDelegationTokens, TestObjectReleaseTracker, TestReversedWildcardFilterFactory, CdcrReplicationDistributedZkTest, HdfsLockFactoryTest, TestJoin, CoreMergeIndexesAdminHandlerTest, ConnectionManagerTest, SearchHandlerTest, TestNRTOpen, TestTrackingShardHandlerFactory, TestMultiValuedNumericRangeQuery, TestCloudInspectUtil, TestConfigsApi, TestBulkSchemaAPI, TestFilteredDocIdSet, TestShortCircuitedRequests]
   [junit4] Completed [529/764 (2!)] on J0 in 635.86s, 1 test, 1 error <<< FAILURES!

[...truncated 44657 lines...]