You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2017/05/19 06:03:17 UTC

[JENKINS] Lucene-Solr-Tests-6.x - Build # 899 - Still Unstable

Build: https://builds.apache.org/job/Lucene-Solr-Tests-6.x/899/

2 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI

Error Message:
expected:<3> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<3> but was:<0>
	at __randomizedtesting.SeedInfo.seed([BDDB8636F82AF7B7:F5AEF282FE19D822]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:522)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.testDelegationTokenRenew

Error Message:
expected:<200> but was:<403>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<403>
	at __randomizedtesting.SeedInfo.seed([BDDB8636F82AF7B7:8A407228C0E62A13]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.renewDelegationToken(TestDelegationWithHadoopAuth.java:118)
	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.verifyDelegationTokenRenew(TestDelegationWithHadoopAuth.java:302)
	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.testDelegationTokenRenew(TestDelegationWithHadoopAuth.java:319)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11344 lines...]
   [junit4] Suite: org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_BDDB8636F82AF7B7-001/init-core-data-001
   [junit4]   2> 284983 WARN  (SUITE-TestDelegationWithHadoopAuth-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=37 numCloses=37
   [junit4]   2> 284984 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 284987 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 284991 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_BDDB8636F82AF7B7-001/tempDir-001
   [junit4]   2> 284991 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 285000 INFO  (Thread-380) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 285000 INFO  (Thread-380) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 285056 ERROR (Thread-380) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 285100 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:43918
   [junit4]   2> 285194 INFO  (jetty-launcher-217-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 285232 INFO  (jetty-launcher-217-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 285240 INFO  (jetty-launcher-217-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@64b91c9f{/solr,null,AVAILABLE}
   [junit4]   2> 285247 INFO  (jetty-launcher-217-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f5b381f{/solr,null,AVAILABLE}
   [junit4]   2> 285257 INFO  (jetty-launcher-217-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5472521e{SSL,[ssl, http/1.1]}{127.0.0.1:59581}
   [junit4]   2> 285257 INFO  (jetty-launcher-217-thread-2) [    ] o.e.j.s.Server Started @288312ms
   [junit4]   2> 285257 INFO  (jetty-launcher-217-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59581}
   [junit4]   2> 285257 ERROR (jetty-launcher-217-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 285257 INFO  (jetty-launcher-217-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@747dac49{SSL,[ssl, http/1.1]}{127.0.0.1:42348}
   [junit4]   2> 285257 INFO  (jetty-launcher-217-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 285257 INFO  (jetty-launcher-217-thread-1) [    ] o.e.j.s.Server Started @288312ms
   [junit4]   2> 285257 INFO  (jetty-launcher-217-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 285257 INFO  (jetty-launcher-217-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 285257 INFO  (jetty-launcher-217-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42348}
   [junit4]   2> 285257 INFO  (jetty-launcher-217-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-19T05:16:19.400Z
   [junit4]   2> 285257 ERROR (jetty-launcher-217-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 285258 INFO  (jetty-launcher-217-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 285258 INFO  (jetty-launcher-217-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 285258 INFO  (jetty-launcher-217-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 285258 INFO  (jetty-launcher-217-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-19T05:16:19.401Z
   [junit4]   2> 285263 INFO  (jetty-launcher-217-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 285263 INFO  (jetty-launcher-217-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 285281 INFO  (jetty-launcher-217-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 285292 INFO  (jetty-launcher-217-thread-2) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 285292 INFO  (jetty-launcher-217-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43918/solr
   [junit4]   2> 285294 INFO  (jetty-launcher-217-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43918/solr
   [junit4]   2> 285343 INFO  (jetty-launcher-217-thread-1) [n:127.0.0.1:42348_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 285344 INFO  (jetty-launcher-217-thread-2) [n:127.0.0.1:59581_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 285345 INFO  (jetty-launcher-217-thread-1) [n:127.0.0.1:42348_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42348_solr
   [junit4]   2> 285346 INFO  (jetty-launcher-217-thread-2) [n:127.0.0.1:59581_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59581_solr
   [junit4]   2> 285346 INFO  (jetty-launcher-217-thread-1) [n:127.0.0.1:42348_solr    ] o.a.s.c.Overseer Overseer (id=97987525289639941-127.0.0.1:42348_solr-n_0000000000) starting
   [junit4]   2> 285350 INFO  (zkCallback-228-thread-1-processing-n:127.0.0.1:42348_solr) [n:127.0.0.1:42348_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 285350 INFO  (zkCallback-229-thread-1-processing-n:127.0.0.1:59581_solr) [n:127.0.0.1:59581_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 285371 INFO  (jetty-launcher-217-thread-1) [n:127.0.0.1:42348_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42348_solr
   [junit4]   2> 285373 INFO  (zkCallback-228-thread-1-processing-n:127.0.0.1:42348_solr) [n:127.0.0.1:42348_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 285373 INFO  (zkCallback-229-thread-1-processing-n:127.0.0.1:59581_solr) [n:127.0.0.1:59581_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 285376 INFO  (jetty-launcher-217-thread-2) [n:127.0.0.1:59581_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 285396 INFO  (jetty-launcher-217-thread-2) [n:127.0.0.1:59581_solr    ] o.a.s.s.HadoopAuthPlugin Params: {delegation-token.token-kind=solr-dt, cookie.domain=127.0.0.1, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, type=simple, zk-dt-secret-manager.enable=true, signer.secret.provider=zookeeper}
   [junit4]   2> 285511 INFO  (jetty-launcher-217-thread-1) [n:127.0.0.1:42348_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 285512 INFO  (jetty-launcher-217-thread-1) [n:127.0.0.1:42348_solr    ] o.a.s.s.HadoopAuthPlugin Params: {delegation-token.token-kind=solr-dt, cookie.domain=127.0.0.1, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, type=simple, zk-dt-secret-manager.enable=true, signer.secret.provider=zookeeper}
   [junit4]   2> 285547 INFO  (jetty-launcher-217-thread-2) [n:127.0.0.1:59581_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 285547 INFO  (jetty-launcher-217-thread-1) [n:127.0.0.1:42348_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 285589 INFO  (jetty-launcher-217-thread-1-EventThread) [n:127.0.0.1:42348_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 285589 INFO  (jetty-launcher-217-thread-2-EventThread) [n:127.0.0.1:59581_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 285821 INFO  (jetty-launcher-217-thread-1) [n:127.0.0.1:42348_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 285822 INFO  (jetty-launcher-217-thread-1) [n:127.0.0.1:42348_solr    ] o.a.s.h.c.HttpShardHandlerFactory Reconfiguring the default client with: org.apache.solr.security.PKIAuthenticationPlugin$HttpHeaderClientConfigurer@1662decd
   [junit4]   2> 285822 INFO  (jetty-launcher-217-thread-1) [n:127.0.0.1:42348_solr    ] o.a.s.u.UpdateShardHandler Reconfiguring the default client with: org.apache.solr.security.PKIAuthenticationPlugin$HttpHeaderClientConfigurer@1662decd
   [junit4]   2> 285824 INFO  (jetty-launcher-217-thread-2) [n:127.0.0.1:59581_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 285824 INFO  (jetty-launcher-217-thread-2) [n:127.0.0.1:59581_solr    ] o.a.s.h.c.HttpShardHandlerFactory Reconfiguring the default client with: org.apache.solr.security.PKIAuthenticationPlugin$HttpHeaderClientConfigurer@4a20a41b
   [junit4]   2> 285825 INFO  (jetty-launcher-217-thread-2) [n:127.0.0.1:59581_solr    ] o.a.s.u.UpdateShardHandler Reconfiguring the default client with: org.apache.solr.security.PKIAuthenticationPlugin$HttpHeaderClientConfigurer@4a20a41b
   [junit4]   2> 285875 INFO  (jetty-launcher-217-thread-1) [n:127.0.0.1:42348_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_BDDB8636F82AF7B7-001/tempDir-001/node1/.
   [junit4]   2> 285875 INFO  (jetty-launcher-217-thread-2) [n:127.0.0.1:59581_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_BDDB8636F82AF7B7-001/tempDir-001/node2/.
   [junit4] IGNOR/A 0.00s J2 | TestDelegationWithHadoopAuth.testDelegationTokenCancelFail
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://issues.apache.org/jira/browse/HADOOP-14044))
   [junit4]   2> 285920 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenCancel
   [junit4]   2> 286466 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 286469 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43918/solr ready
   [junit4]   2> 286486 ERROR (qtp472651473-1902) [    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_1]
   [junit4]   2> 286515 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 286516 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43918/solr ready
   [junit4]   2> 286542 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[BDDB8636F82AF7B7]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= , 01-1-45 00:00:00 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 01-1-45 00:00:00 GMT
   [junit4]   2> 286555 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 286556 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43918/solr ready
   [junit4]   2> 286566 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[BDDB8636F82AF7B7]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= , 01-1-45 00:00:00 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 01-1-45 00:00:00 GMT
   [junit4]   2> 286654 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 286655 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43918/solr ready
   [junit4]   2> 286659 ERROR (qtp472651473-1891) [    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_2]
   [junit4]   2> 286681 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[BDDB8636F82AF7B7]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= , 01-1-45 00:00:00 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 01-1-45 00:00:00 GMT
   [junit4]   2> 286705 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[BDDB8636F82AF7B7]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= , 01-1-45 00:00:00 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 01-1-45 00:00:00 GMT
   [junit4]   2> 286712 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenCancel
   [junit4]   2> 286736 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenRenewFail
   [junit4]   2> 286747 WARN  (qtp472651473-1897) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: bar tries to renew a token with renewer foo
   [junit4]   2> 286747 WARN  (qtp472651473-1897) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 286748 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[BDDB8636F82AF7B7]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= , 01-1-45 00:00:00 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 01-1-45 00:00:00 GMT
   [junit4]   2> 286750 WARN  (qtp1288841218-1887) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: bar tries to renew a token with renewer foo
   [junit4]   2> 286751 WARN  (qtp1288841218-1887) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 286751 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[BDDB8636F82AF7B7]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= , 01-1-45 00:00:00 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 01-1-45 00:00:00 GMT
   [junit4]   2> 286762 WARN  (qtp472651473-1891) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: notRenewUser tries to renew a token with renewer renewUser
   [junit4]   2> 286762 WARN  (qtp472651473-1891) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 286763 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[BDDB8636F82AF7B7]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= , 01-1-45 00:00:00 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 01-1-45 00:00:00 GMT
   [junit4]   2> 286765 WARN  (qtp1288841218-1881) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: notRenewUser tries to renew a token with renewer renewUser
   [junit4]   2> 286765 WARN  (qtp1288841218-1881) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 286766 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[BDDB8636F82AF7B7]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= , 01-1-45 00:00:00 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 01-1-45 00:00:00 GMT
   [junit4]   2> 286766 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenRenewFail
   [junit4]   2> 286793 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationOtherOp-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationOtherOp
   [junit4]   2> 286805 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationOtherOp-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 286806 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationOtherOp-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43918/solr ready
   [junit4]   2> 286816 INFO  (qtp472651473-1892) [n:127.0.0.1:59581_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params op=someSolrOperation&action=LIST&user.name=foo&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 286816 INFO  (qtp472651473-1892) [n:127.0.0.1:59581_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={op=someSolrOperation&action=LIST&user.name=foo&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 286820 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationOtherOp-seed#[BDDB8636F82AF7B7]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=foo&p=foo&t=simple-dt&e=1495206980953&s=ycx3YIgywhpJYMHEabR3hw0BmfI="; Domain=127.0.0.1; Expires= , 19-5-29 15:16:20 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 19-5-29 15:16:20 GMT
   [junit4]   2> 286829 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationOtherOp-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationOtherOp
   [junit4]   2> 286846 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenVerify
   [junit4]   2> 286894 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 286895 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43918/solr ready
   [junit4]   2> 286900 INFO  (qtp472651473-1890) [n:127.0.0.1:59581_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVwfIpx_igFcQy8gfwUEFG7sqQ-wlZU_HvvtgAp44qRxySULB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 286900 INFO  (qtp472651473-1890) [n:127.0.0.1:59581_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVwfIpx_igFcQy8gfwUEFG7sqQ-wlZU_HvvtgAp44qRxySULB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 286927 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[BDDB8636F82AF7B7]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= , 01-1-45 00:00:00 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 01-1-45 00:00:00 GMT
   [junit4]   2> 286943 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 286944 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43918/solr ready
   [junit4]   2> 286955 INFO  (qtp472651473-1902) [n:127.0.0.1:59581_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVwfIpx_igFcQy8gfwUEFG7sqQ-wlZU_HvvtgAp44qRxySULB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 286956 INFO  (qtp472651473-1902) [n:127.0.0.1:59581_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVwfIpx_igFcQy8gfwUEFG7sqQ-wlZU_HvvtgAp44qRxySULB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 286982 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 286983 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43918/solr ready
   [junit4]   2> 286996 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[BDDB8636F82AF7B7]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= , 01-1-45 00:00:00 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 01-1-45 00:00:00 GMT
   [junit4]   2> 286997 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenVerify
   [junit4]   2> 287022 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenRenew
   [junit4]   2> 287032 WARN  (qtp472651473-1902) [n:127.0.0.1:59581_solr    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: foo is trying to renew a token with wrong password
   [junit4]   2> 287033 WARN  (qtp472651473-1902) [n:127.0.0.1:59581_solr    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 287033 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[BDDB8636F82AF7B7]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= , 01-1-45 00:00:00 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 01-1-45 00:00:00 GMT
   [junit4]   2> 287033 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenRenew
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDelegationWithHadoopAuth -Dtests.method=testDelegationTokenRenew -Dtests.seed=BDDB8636F82AF7B7 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ja-JP-u-ca-japanese-x-lvariant-JP -Dtests.timezone=Etc/Zulu -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.04s J2 | TestDelegationWithHadoopAuth.testDelegationTokenRenew <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<200> but was:<403>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BDDB8636F82AF7B7:8A407228C0E62A13]:0)
   [junit4]    > 	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.renewDelegationToken(TestDelegationWithHadoopAuth.java:118)
   [junit4]    > 	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.verifyDelegationTokenRenew(TestDelegationWithHadoopAuth.java:302)
   [junit4]    > 	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.testDelegationTokenRenew(TestDelegationWithHadoopAuth.java:319)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 287056 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenSolrClient-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenSolrClient
   [junit4]   2> 287070 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenSolrClient-seed#[BDDB8636F82AF7B7]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= , 01-1-45 00:00:00 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 01-1-45 00:00:00 GMT
   [junit4]   2> 287078 INFO  (qtp472651473-1902) [n:127.0.0.1:59581_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVwfIp1RigFcQy8hUQcEFBbjPaku4YxJS60-HnxSvAPBwnXEB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 287078 INFO  (qtp472651473-1902) [n:127.0.0.1:59581_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVwfIp1RigFcQy8hUQcEFBbjPaku4YxJS60-HnxSvAPBwnXEB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 287079 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenSolrClient-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenSolrClient
   [junit4]   2> 287100 INFO  (TEST-TestDelegationWithHadoopAuth.testZNodePaths-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testZNodePaths
   [junit4]   2> 287122 INFO  (TEST-TestDelegationWithHadoopAuth.testZNodePaths-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testZNodePaths
   [junit4]   2> 287128 INFO  (jetty-closer-218-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@5472521e{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 287129 INFO  (jetty-closer-218-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=860681489
   [junit4]   2> 287129 INFO  (jetty-closer-218-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@747dac49{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 287129 INFO  (jetty-closer-218-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 287129 INFO  (jetty-closer-218-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1269566974
   [junit4]   2> 287129 INFO  (jetty-closer-218-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jvm
   [junit4]   2> 287130 INFO  (jetty-closer-218-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jetty
   [junit4]   2> 287130 INFO  (jetty-closer-218-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 287130 INFO  (jetty-closer-218-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jvm
   [junit4]   2> 287130 INFO  (jetty-closer-218-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jetty
   [junit4]   2> 287132 INFO  (jetty-closer-218-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 287132 INFO  (jetty-closer-218-thread-2) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 287136 INFO  (jetty-closer-218-thread-2) [    ] o.a.s.c.Overseer Overseer (id=97987525289639941-127.0.0.1:42348_solr-n_0000000000) closing
   [junit4]   2> 287137 INFO  (OverseerStateUpdate-97987525289639941-127.0.0.1:42348_solr-n_0000000000) [n:127.0.0.1:42348_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:42348_solr
   [junit4]   2> 287137 WARN  (zkCallback-229-thread-1-processing-n:127.0.0.1:59581_solr) [n:127.0.0.1:59581_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 287138 INFO  (zkCallback-228-thread-1-processing-n:127.0.0.1:42348_solr) [n:127.0.0.1:42348_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 287140 ERROR (Thread[Thread-402,5,TGRP-TestDelegationWithHadoopAuth]) [n:127.0.0.1:59581_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 287140 WARN  (zkCallback-228-thread-1-processing-n:127.0.0.1:42348_solr) [n:127.0.0.1:42348_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 287141 ERROR (Thread[Thread-404,5,TGRP-TestDelegationWithHadoopAuth]) [n:127.0.0.1:42348_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 287158 INFO  (jetty-closer-218-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4f5b381f{/solr,null,UNAVAILABLE}
   [junit4]   2> 287159 INFO  (jetty-closer-218-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@64b91c9f{/solr,null,UNAVAILABLE}
   [junit4]   2> 287162 ERROR (SUITE-TestDelegationWithHadoopAuth-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 287162 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43918 43918
   [junit4]   2> 287263 INFO  (Thread-380) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43918 43918
   [junit4]   2> 287265 WARN  (Thread-380) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	10	/solr/aliases.json
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	10	/solr/clusterprops.json
   [junit4]   2> 	10	/solr/clusterstate.json
   [junit4]   2> 	3	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_7
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_6
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_8
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_5
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	10	/solr/live_nodes
   [junit4]   2> 	10	/solr/collections
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
   [junit4]   2> 
   [junit4]   2> 287265 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J2/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_BDDB8636F82AF7B7-001
   [junit4]   2> May 19, 2017 5:16:21 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=true,coord=yes): {}, locale=ja-JP-u-ca-japanese-x-lvariant-JP, timezone=Etc/Zulu
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=168935584,total=512753664
   [junit4]   2> NOTE: All tests run in this JVM: [TestMacros, TestFieldCacheSort, StatelessScriptUpdateProcessorFactoryTest, BlockJoinFacetSimpleTest, TermVectorComponentDistributedTest, HLLUtilTest, TestShortCircuitedRequests, MultiThreadedOCPTest, TestComplexPhraseQParserPlugin, PropertiesRequestHandlerTest, TestSolrXml, TestRestoreCore, TestExpandComponent, TestBulkSchemaConcurrent, TestPostingsSolrHighlighter, ExplicitHLLTest, DateRangeFieldTest, TestMinMaxOnMultiValuedField, TestInPlaceUpdatesStandalone, HttpPartitionTest, TestDelegationWithHadoopAuth]
   [junit4] Completed [84/712 (1!)] on J2 in 3.09s, 8 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 101 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_BDDB8636F82AF7B7-001/init-core-data-001
   [junit4]   2> 297737 WARN  (SUITE-CollectionsAPIDistributedZkTest-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 297738 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 297739 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 297740 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_BDDB8636F82AF7B7-001/tempDir-001
   [junit4]   2> 297740 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 297741 INFO  (Thread-508) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 297741 INFO  (Thread-508) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 297749 ERROR (Thread-508) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 297841 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:38479
   [junit4]   2> 297865 INFO  (jetty-launcher-390-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 297868 INFO  (jetty-launcher-390-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 297869 INFO  (jetty-launcher-390-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@15218583{/solr,null,AVAILABLE}
   [junit4]   2> 297870 INFO  (jetty-launcher-390-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@10b7ee93{SSL,[ssl, http/1.1]}{127.0.0.1:42610}
   [junit4]   2> 297870 INFO  (jetty-launcher-390-thread-1) [    ] o.e.j.s.Server Started @301100ms
   [junit4]   2> 297870 INFO  (jetty-launcher-390-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42610}
   [junit4]   2> 297870 ERROR (jetty-launcher-390-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 297871 INFO  (jetty-launcher-390-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 297871 INFO  (jetty-launcher-390-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 297871 INFO  (jetty-launcher-390-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 297871 INFO  (jetty-launcher-390-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-19T05:16:32.184Z
   [junit4]   2> 297873 INFO  (jetty-launcher-390-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 297877 INFO  (jetty-launcher-390-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 297882 INFO  (jetty-launcher-390-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4e035d5c{/solr,null,AVAILABLE}
   [junit4]   2> 297883 INFO  (jetty-launcher-390-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5d663827{SSL,[ssl, http/1.1]}{127.0.0.1:57025}
   [junit4]   2> 297883 INFO  (jetty-launcher-390-thread-2) [    ] o.e.j.s.Server Started @301113ms
   [junit4]   2> 297883 INFO  (jetty-launcher-390-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=57025}
   [junit4]   2> 297883 ERROR (jetty-launcher-390-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 297884 INFO  (jetty-launcher-390-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 297884 INFO  (jetty-launcher-390-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 297884 INFO  (jetty-launcher-390-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 297884 INFO  (jetty-launcher-390-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-19T05:16:32.197Z
   [junit4]   2> 297886 INFO  (jetty-launcher-390-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7f71bf80{/solr,null,AVAILABLE}
   [junit4]   2> 297887 INFO  (jetty-launcher-390-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4aad29d2{SSL,[ssl, http/1.1]}{127.0.0.1:46882}
   [junit4]   2> 297887 INFO  (jetty-launcher-390-thread-3) [    ] o.e.j.s.Server Started @301117ms
   [junit4]   2> 297887 INFO  (jetty-launcher-390-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46882}
   [junit4]   2> 297887 ERROR (jetty-launcher-390-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 297887 INFO  (jetty-launcher-390-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 297887 INFO  (jetty-launcher-390-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 297887 INFO  (jetty-launcher-390-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 297887 INFO  (jetty-launcher-390-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-19T05:16:32.200Z
   [junit4]   2> 297898 INFO  (jetty-launcher-390-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@54ff3b75{/solr,null,AVAILABLE}
   [junit4]   2> 297899 INFO  (jetty-launcher-390-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@671e2e02{SSL,[ssl, http/1.1]}{127.0.0.1:55619}
   [junit4]   2> 297899 INFO  (jetty-launcher-390-thread-4) [    ] o.e.j.s.Server Started @301129ms
   [junit4]   2> 297899 INFO  (jetty-launcher-390-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55619}
   [junit4]   2> 297899 ERROR (jetty-launcher-390-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 297899 INFO  (jetty-launcher-390-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 297900 INFO  (jetty-launcher-390-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 297900 INFO  (jetty-launcher-390-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 297900 INFO  (jetty-launcher-390-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-19T05:16:32.213Z
   [junit4]   2> 297902 INFO  (jetty-launcher-390-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 297920 INFO  (jetty-launcher-390-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 297925 INFO  (jetty-launcher-390-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 297930 INFO  (jetty-launcher-390-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 297930 INFO  (jetty-launcher-390-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 297931 INFO  (jetty-launcher-390-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38479/solr
   [junit4]   2> 297931 INFO  (jetty-launcher-390-thread-3) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 297933 INFO  (jetty-launcher-390-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38479/solr
   [junit4]   2> 297937 INFO  (jetty-launcher-390-thread-4) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 297945 INFO  (jetty-launcher-390-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38479/solr
   [junit4]   2> 297948 INFO  (jetty-launcher-390-thread-2) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 297955 INFO  (jetty-launcher-390-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38479/solr
   [junit4]   2> 297956 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15c1f22c7bb0007, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 297981 INFO  (jetty-launcher-390-thread-3) [n:127.0.0.1:46882_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 297982 INFO  (jetty-launcher-390-thread-1) [n:127.0.0.1:42610_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 297982 INFO  (jetty-launcher-390-thread-4) [n:127.0.0.1:55619_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 297982 INFO  (jetty-launcher-390-thread-3) [n:127.0.0.1:46882_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46882_solr
   [junit4]   2> 297983 INFO  (jetty-launcher-390-thread-1) [n:127.0.0.1:42610_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42610_solr
   [junit4]   2> 297984 INFO  (jetty-launcher-390-thread-4) [n:127.0.0.1:55619_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55619_solr
   [junit4]   2> 297984 INFO  (jetty-launcher-390-thread-3) [n:127.0.0.1:46882_solr    ] o.a.s.c.Overseer Overseer (id=97987526135578632-127.0.0.1:46882_solr-n_0000000000) starting
   [junit4]   2> 297987 INFO  (zkCallback-408-thread-1-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 297994 INFO  (zkCallback-407-thread-1-processing-n:127.0.0.1:46882_solr) [n:127.0.0.1:46882_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 297998 INFO  (zkCallback-411-thread-1-processing-n:127.0.0.1:55619_solr) [n:127.0.0.1:55619_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 298006 INFO  (jetty-launcher-390-thread-2) [n:127.0.0.1:57025_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 298008 INFO  (jetty-launcher-390-thread-2) [n:127.0.0.1:57025_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 298010 INFO  (jetty-launcher-390-thread-2) [n:127.0.0.1:57025_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57025_solr
   [junit4]   2> 298011 INFO  (zkCallback-408-thread-1-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 298011 INFO  (zkCallback-411-thread-1-processing-n:127.0.0.1:55619_solr) [n:127.0.0.1:55619_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 298012 INFO  (zkCallback-407-thread-1-processing-n:127.0.0.1:46882_solr) [n:127.0.0.1:46882_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 298022 INFO  (zkCallback-412-thread-1-processing-n:127.0.0.1:57025_solr) [n:127.0.0.1:57025_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 298039 INFO  (jetty-launcher-390-thread-3) [n:127.0.0.1:46882_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46882_solr
   [junit4]   2> 298040 INFO  (zkCallback-411-thread-1-processing-n:127.0.0.1:55619_solr) [n:127.0.0.1:55619_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 298040 INFO  (zkCallback-412-thread-1-processing-n:127.0.0.1:57025_solr) [n:127.0.0.1:57025_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 298041 INFO  (zkCallback-408-thread-1-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 298041 INFO  (zkCallback-407-thread-1-processing-n:127.0.0.1:46882_solr) [n:127.0.0.1:46882_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 298156 INFO  (jetty-launcher-390-thread-4) [n:127.0.0.1:55619_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_BDDB8636F82AF7B7-001/tempDir-001/node4/.
   [junit4]   2> 298164 INFO  (jetty-launcher-390-thread-2) [n:127.0.0.1:57025_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_BDDB8636F82AF7B7-001/tempDir-001/node2/.
   [junit4]   2> 298169 INFO  (jetty-launcher-390-thread-3) [n:127.0.0.1:46882_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_BDDB8636F82AF7B7-001/tempDir-001/node3/.
   [junit4]   2> 298215 INFO  (jetty-launcher-390-thread-1) [n:127.0.0.1:42610_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_BDDB8636F82AF7B7-001/tempDir-001/node1/.
   [junit4]   2> 298228 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 298229 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38479/solr ready
   [junit4]   2> 298275 INFO  (TEST-CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDeleteNonExistentCollection
   [junit4]   2> 298276 INFO  (TEST-CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 298342 INFO  (qtp692328705-2222) [n:127.0.0.1:55619_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=unknown_collection&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 298345 INFO  (OverseerThreadFactory-923-thread-1-processing-n:127.0.0.1:46882_solr) [n:127.0.0.1:46882_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 298346 ERROR (OverseerThreadFactory-923-thread-1-processing-n:127.0.0.1:46882_solr) [n:127.0.0.1:46882_solr    ] o.a.s.c.OverseerCollectionMessageHandler Collection: unknown_collection operation: delete failed:org.apache.solr.common.SolrException: Could not find collection : unknown_collection
   [junit4]   2> 	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:194)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionMessageHandler.collectionCmd(OverseerCollectionMessageHandler.java:846)
   [junit4]   2> 	at org.apache.solr.cloud.DeleteCollectionCmd.call(DeleteCollectionCmd.java:88)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:226)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:464)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 298381 INFO  (qtp692328705-2222) [n:127.0.0.1:55619_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=unknown_collection&action=DELETE&wt=javabin&version=2} status=400 QTime=39
   [junit4]   2> 298385 INFO  (qtp692328705-2223) [n:127.0.0.1:55619_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&name=acollectionafterbaddelete&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 298387 INFO  (OverseerThreadFactory-923-thread-2-processing-n:127.0.0.1:46882_solr) [n:127.0.0.1:46882_solr    ] o.a.s.c.CreateCollectionCmd Create collection acollectionafterbaddelete
   [junit4]   2> 298388 INFO  (OverseerCollectionConfigSetProcessor-97987526135578632-127.0.0.1:46882_solr-n_0000000000) [n:127.0.0.1:46882_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 298564 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=acollectionafterbaddelete_shard1_replica2&action=CREATE&numShards=1&collection=acollectionafterbaddelete&shard=shard1&wt=javabin&version=2
   [junit4]   2> 298565 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 298570 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=acollectionafterbaddelete_shard1_replica1&action=CREATE&numShards=1&collection=acollectionafterbaddelete&shard=shard1&wt=javabin&version=2
   [junit4]   2> 298570 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 298676 INFO  (zkCallback-412-thread-1-processing-n:127.0.0.1:57025_solr) [n:127.0.0.1:57025_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 298676 INFO  (zkCallback-408-thread-1-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 299582 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 299590 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 299591 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.s.IndexSchema [acollectionafterbaddelete_shard1_replica1] Schema name=minimal
   [junit4]   2> 299594 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 299594 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'acollectionafterbaddelete_shard1_replica1' using configuration from collection acollectionafterbaddelete, trusted=true
   [junit4]   2> 299595 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 299595 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.SolrCore [[acollectionafterbaddelete_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_BDDB8636F82AF7B7-001/tempDir-001/node2/acollectionafterbaddelete_shard1_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_BDDB8636F82AF7B7-001/tempDir-001/node2/./acollectionafterbaddelete_shard1_replica1/data/]
   [junit4]   2> 299600 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.s.IndexSchema [acollectionafterbaddelete_shard1_replica2] Schema name=minimal
   [junit4]   2> 299602 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 299602 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'acollectionafterbaddelete_shard1_replica2' using configuration from collection acollectionafterbaddelete, trusted=true
   [junit4]   2> 299603 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 299603 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.SolrCore [[acollectionafterbaddelete_shard1_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_BDDB8636F82AF7B7-001/tempDir-001/node1/acollectionafterbaddelete_shard1_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_BDDB8636F82AF7B7-001/tempDir-001/node1/./acollectionafterbaddelete_shard1_replica2/data/]
   [junit4]   2> 299645 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 299645 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 299647 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 299647 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 299648 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@e1fda89[acollectionafterbaddelete_shard1_replica1] main]
   [junit4]   2> 299650 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 299650 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 299650 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 299650 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1567800420165746688
   [junit4]   2> 299653 INFO  (searcherExecutor-932-thread-1-processing-n:127.0.0.1:57025_solr x:acollectionafterbaddelete_shard1_replica1 s:shard1 c:acollectionafterbaddelete) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.SolrCore [acollectionafterbaddelete_shard1_replica1] Registered new searcher Searcher@e1fda89[acollectionafterbaddelete_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 299658 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 299664 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 299664 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 299665 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 299665 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 299666 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@d3fe185[acollectionafterbaddelete_shard1_replica2] main]
   [junit4]   2> 299667 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 299667 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 299667 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 299668 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1567800420184621056
   [junit4]   2> 299670 INFO  (searcherExecutor-933-thread-1-processing-n:127.0.0.1:42610_solr x:acollectionafterbaddelete_shard1_replica2 s:shard1 c:acollectionafterbaddelete) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.SolrCore [acollectionafterbaddelete_shard1_replica2] Registered new searcher Searcher@d3fe185[acollectionafterbaddelete_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 299761 INFO  (zkCallback-412-thread-1-processing-n:127.0.0.1:57025_solr) [n:127.0.0.1:57025_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 299761 INFO  (zkCallback-408-thread-1-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 300159 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 300159 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 300159 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:57025/solr/acollectionafterbaddelete_shard1_replica1/
   [junit4]   2> 300159 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.u.PeerSync PeerSync: core=acollectionafterbaddelete_shard1_replica1 url=https://127.0.0.1:57025/solr START replicas=[https://127.0.0.1:42610/solr/acollectionafterbaddelete_shard1_replica2/] nUpdates=100
   [junit4]   2> 300159 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.u.PeerSync PeerSync: core=acollectionafterbaddelete_shard1_replica1 url=https://127.0.0.1:57025/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 300183 INFO  (qtp1870289826-2183) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1 r:core_node1 x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.S.Request [acollectionafterbaddelete_shard1_replica2]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 300190 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 300190 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 300190 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 300194 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:57025/solr/acollectionafterbaddelete_shard1_replica1/ shard1
   [junit4]   2> 300297 INFO  (zkCallback-412-thread-1-processing-n:127.0.0.1:57025_solr) [n:127.0.0.1:57025_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 300297 INFO  (zkCallback-408-thread-1-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 300345 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 300348 INFO  (qtp1388972491-2196) [n:127.0.0.1:57025_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=acollectionafterbaddelete_shard1_replica1&action=CREATE&numShards=1&collection=acollectionafterbaddelete&shard=shard1&wt=javabin&version=2} status=0 QTime=1778
   [junit4]   2> 300450 INFO  (zkCallback-408-thread-1-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 300450 INFO  (zkCallback-412-thread-1-processing-n:127.0.0.1:57025_solr) [n:127.0.0.1:57025_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 300676 INFO  (qtp1870289826-2184) [n:127.0.0.1:42610_solr c:acollectionafterbaddelete s:shard1  x:acollectionafterbaddelete_shard1_replica2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=acollectionafterbaddelete_shard1_replica2&action=CREATE&numShards=1&collection=acollectionafterbaddelete&shard=shard1&wt=javabin&version=2} status=0 QTime=2112
   [junit4]   2> 300683 INFO  (qtp692328705-2223) [n:127.0.0.1:55619_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 300779 INFO  (zkCallback-412-thread-1-processing-n:127.0.0.1:57025_solr) [n:127.0.0.1:57025_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 300779 INFO  (zkCallback-408-thread-1-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 301685 INFO  (qtp692328705-2223) [n:127.0.0.1:55619_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&collection.configName=conf&name=acollectionafterbaddelete&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=3299
   [junit4]   2> 301690 INFO  (TEST-CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDeleteNonExistentCollection
   [junit4]   2> 301721 INFO  (TEST-CollectionsAPIDistributedZkTest.testNoCollectionSpecified-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNoCollectionSpecified
   [junit4]   2> 301723 INFO  (TEST-CollectionsAPIDistributedZkTest.testNoCollectionSpecified-seed#[BDDB8636F82AF7B7]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 301726 INFO  (qtp692328705-2216) [n:127.0.0.1:55619_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=acollectionafterbaddelete&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 301729 INFO  (OverseerCollectionConfigSetProcessor-97987526135578632-127.0.0.1:46882_solr-n_0000000000) [n:127.0.0.1:46882_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 301730 INFO  (OverseerThreadFactory-923-thread-3-processing-n:127.0.0.1:46882_solr) [n:127.0.0.1:46882_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 301733 INFO  (qtp1388972491-2197) [n:127.0.0.1:57025_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.acollectionafterbaddelete.shard1.replica1
   [junit4]   2> 301733 INFO  (qtp1870289826-2185) [n:127.0.0.1:42610_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.acollectionafterbaddelete.shard1.replica2
   [junit4]   2> 301734 INFO  (qtp1388972491-2197) [n:127.0.0.1:57025_solr    ] o.a.s.c.SolrCore [acollectionafterbaddelete_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@503d5f5
   [junit4]   2> 301734 INFO  (qtp1870289826-2185) [n:127.0.0.1:42610_solr    ] o.a.s.c.SolrCore [acollectionafterbaddelete_shard1_replica2]  CLOSING SolrCore org.apache.solr.core.SolrCore@395216ec
   [junit4]   2> 301738 INFO  (qtp1388972491-2197) [n:127.0.0.1:57025_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.acollectionafterbaddelete.shard1.replica1
   [junit4]   2> 301738 INFO  (qtp1870289826-2185) [n:127.0.0.1:42610_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.acollectionafterbaddelete.shard1.replica2
   [junit4]   2> 301740 INFO  (qtp1870289826-2185) [n:127.0.0.1:42610_solr    ] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
   [junit4]   2> 301740 WARN  (zkCallback-408-thread-1-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.LeaderElector Our node is no longer in line to be leader
   [junit4]   2> 301741 INFO  (qtp1870289826-2185) [n:127.0.0.1:42610_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=acollectionafterbaddelete_shard1_replica2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=8
   [junit4]   2> 301742 INFO  (qtp1388972491-2197) [n:127.0.0.1:57025_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=acollectionafterbaddelete_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=8
   [junit4]   2> 302449 INFO  (qtp692328705-2216) [n:127.0.0.1:55619_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=acollectionafterbaddelete&action=DELETE&wt=javabin&version=2} status=0 QTime=723
   [junit4]   2> 302453 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr    ] o.a.s.h.a.CoreAdminOperation core create command dataDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_BDDB8636F82AF7B7-001/tempDir-002&collection.configName=conf&name=corewithnocollection&action=CREATE&numShards=1&collection=&wt=javabin&version=2
   [junit4]   2> 302454 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 302455 INFO  (OverseerStateUpdate-97987526135578632-127.0.0.1:46882_solr-n_0000000000) [n:127.0.0.1:46882_solr    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 303501 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 303509 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.s.IndexSchema [corewithnocollection] Schema name=minimal
   [junit4]   2> 303512 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 303512 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.c.CoreContainer Creating SolrCore 'corewithnocollection' using configuration from collection corewithnocollection, trusted=true
   [junit4]   2> 303513 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 303513 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.c.SolrCore [[corewithnocollection] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_BDDB8636F82AF7B7-001/tempDir-001/node4/corewithnocollection], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_BDDB8636F82AF7B7-001/tempDir-002/]
   [junit4]   2> 303594 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 303594 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 303596 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 303596 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 303599 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.s.SolrIndexSearcher Opening [Searcher@3ba4db80[corewithnocollection] main]
   [junit4]   2> 303600 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 303601 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 303601 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 303602 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1567800424309719040
   [junit4]   2> 303602 INFO  (searcherExecutor-942-thread-1-processing-n:127.0.0.1:55619_solr x:corewithnocollection c:corewithnocollection) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.c.SolrCore [corewithnocollection] Registered new searcher Searcher@3ba4db80[corewithnocollection] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 303618 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 303618 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 303618 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:55619/solr/corewithnocollection/
   [junit4]   2> 303618 INFO  (qtp692328705-2219) [n:127.0.0.1:55619_solr c:corewithnocollection   x:corewithnocollection] o.a.s.c.SyncSt

[...truncated too long message...]

 Closing metric reporters for: solr.node
   [junit4]   2> 407707 INFO  (jetty-closer-391-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jvm
   [junit4]   2> 407707 INFO  (jetty-closer-391-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jetty
   [junit4]   2> 407707 INFO  (jetty-closer-391-thread-3) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=560505254
   [junit4]   2> 407707 INFO  (jetty-closer-391-thread-3) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 407707 INFO  (jetty-closer-391-thread-3) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jvm
   [junit4]   2> 407707 INFO  (jetty-closer-391-thread-3) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jetty
   [junit4]   2> 407710 INFO  (jetty-closer-391-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@671e2e02{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 407720 INFO  (jetty-closer-391-thread-3) [    ] o.a.s.c.Overseer Overseer (id=97987526135578632-127.0.0.1:46882_solr-n_0000000000) closing
   [junit4]   2> 407720 INFO  (OverseerStateUpdate-97987526135578632-127.0.0.1:46882_solr-n_0000000000) [n:127.0.0.1:46882_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:46882_solr
   [junit4]   2> 407720 INFO  (jetty-closer-391-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2005915782
   [junit4]   2> 407721 INFO  (jetty-closer-391-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 407721 INFO  (jetty-closer-391-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jvm
   [junit4]   2> 407721 INFO  (jetty-closer-391-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jetty
   [junit4]   2> 407723 WARN  (zkCallback-407-thread-7-processing-n:127.0.0.1:46882_solr) [n:127.0.0.1:46882_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 407725 INFO  (jetty-closer-391-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7f71bf80{/solr,null,UNAVAILABLE}
   [junit4]   2> 407726 INFO  (zkCallback-411-thread-1-processing-n:127.0.0.1:55619_solr) [n:127.0.0.1:55619_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 407726 INFO  (zkCallback-412-thread-6-processing-n:127.0.0.1:57025_solr) [n:127.0.0.1:57025_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 407726 INFO  (zkCallback-408-thread-8-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 407726 INFO  (zkCallback-408-thread-3-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42610_solr
   [junit4]   2> 407728 INFO  (zkCallback-408-thread-3-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.Overseer Overseer (id=97987526135578633-127.0.0.1:42610_solr-n_0000000001) starting
   [junit4]   2> 407738 INFO  (jetty-closer-391-thread-4) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@10b7ee93{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 407740 INFO  (jetty-closer-391-thread-4) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1710092671
   [junit4]   2> 407740 INFO  (jetty-closer-391-thread-4) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 407740 INFO  (jetty-closer-391-thread-4) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jvm
   [junit4]   2> 407740 INFO  (jetty-closer-391-thread-4) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jetty
   [junit4]   2> 407741 WARN  (zkCallback-412-thread-6-processing-n:127.0.0.1:57025_solr) [n:127.0.0.1:57025_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 407742 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:38479) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15c1f22c7bb000c, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 407742 INFO  (jetty-closer-391-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4e035d5c{/solr,null,UNAVAILABLE}
   [junit4]   2> 407743 INFO  (zkCallback-411-thread-1-processing-n:127.0.0.1:55619_solr) [n:127.0.0.1:55619_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 407743 INFO  (zkCallback-408-thread-3-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 407757 INFO  (zkCallback-408-thread-3-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 407760 INFO  (jetty-closer-391-thread-4) [    ] o.a.s.c.Overseer Overseer (id=97987526135578633-127.0.0.1:42610_solr-n_0000000001) closing
   [junit4]   2> 407760 INFO  (OverseerStateUpdate-97987526135578633-127.0.0.1:42610_solr-n_0000000001) [n:127.0.0.1:42610_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:42610_solr
   [junit4]   2> 409255 WARN  (zkCallback-411-thread-1-processing-n:127.0.0.1:55619_solr) [n:127.0.0.1:55619_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 409255 INFO  (jetty-closer-391-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@54ff3b75{/solr,null,UNAVAILABLE}
   [junit4]   2> 409264 WARN  (zkCallback-408-thread-3-processing-n:127.0.0.1:42610_solr) [n:127.0.0.1:42610_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 409265 INFO  (jetty-closer-391-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@15218583{/solr,null,UNAVAILABLE}
   [junit4]   2> 409266 ERROR (SUITE-CollectionsAPIDistributedZkTest-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 409267 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38479 38479
   [junit4]   2> 409411 INFO  (Thread-551) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38479 38479
   [junit4]   2> 409412 WARN  (Thread-551) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	7	/solr/configs/conf
   [junit4]   2> 	3	/solr/aliases.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	57	/solr/clusterstate.json
   [junit4]   2> 	4	/solr/collections/reloaded_collection/state.json
   [junit4]   2> 	3	/solr/clusterprops.json
   [junit4]   2> 	2	/solr/collections/halfdeletedcollection/state.json
   [junit4]   2> 	2	/solr/collections/awhollynewcollection_0/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	133	/solr/collections
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 
   [junit4]   2> 409412 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[BDDB8636F82AF7B7]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_BDDB8636F82AF7B7-001
   [junit4]   2> May 19, 2017 5:18:23 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {_version_=FSTOrd50, id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128)))}, docValues:{}, maxPointsInLeafNode=1753, maxMBSortInHeap=7.811028443423819, sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=ro, timezone=Europe/San_Marino
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=162248344,total=522190848
   [junit4]   2> NOTE: All tests run in this JVM: [TestQueryTypes, ForceLeaderTest, AtomicUpdatesTest, TestMissingGroups, TestExportWriter, TermsComponentTest, SharedFSAutoReplicaFailoverUtilsTest, ConnectionReuseTest, TestElisionMultitermQuery, ConvertedLegacyTest, TestCloudPivotFacet, InfixSuggestersTest, HdfsBasicDistributedZkTest, TestBlendedInfixSuggestions, SolrCoreMetricManagerTest, MoveReplicaTest, TestSubQueryTransformer, TestFieldResource, SolrIndexSplitterTest, CdcrBootstrapTest, IndexSchemaRuntimeFieldTest, JsonLoaderTest, HdfsThreadLeakTest, TestJavabinTupleStreamParser, TestInitParams, TestTrie, TestSolr4Spatial2, TestDynamicFieldCollectionResource, TestSortByMinMaxFunction, TestTrackingShardHandlerFactory, SubstringBytesRefFilterTest, CoreAdminRequestStatusTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed [117/712 (2!)] on J1 in 112.19s, 20 tests, 1 failure <<< FAILURES!

[...truncated 45390 lines...]

[JENKINS] Lucene-Solr-Tests-6.x - Build # 901 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-6.x/901/

All tests passed

Build Log:
[...truncated 20879 lines...]
-validate-source-patterns:
[source-patterns] tabs instead spaces: solr/solr-ref-guide/src/css/ref-guide.css
[source-patterns] tabs instead spaces: solr/solr-ref-guide/src/js/ref-guide-toc.js

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/build.xml:775: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/build.xml:130: Found 2 violations in source files (tabs instead spaces).

Total time: 68 minutes 41 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-Tests-6.x - Build # 900 - Failure

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-6.x/900/

All tests passed

Build Log:
[...truncated 20880 lines...]
-validate-source-patterns:
[source-patterns] tabs instead spaces: solr/solr-ref-guide/src/css/ref-guide.css
[source-patterns] tabs instead spaces: solr/solr-ref-guide/src/js/ref-guide-toc.js

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/build.xml:775: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/build.xml:130: Found 2 violations in source files (tabs instead spaces).

Total time: 69 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any