You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2017/10/07 23:49:37 UTC

[JENKINS] Lucene-Solr-7.x-Linux (32bit/jdk1.8.0_144) - Build # 566 - Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/566/
Java: 32bit/jdk1.8.0_144 -client -XX:+UseSerialGC

2 tests failed.
FAILED:  org.apache.lucene.spatial.spatial4j.Geo3dRptTest.testOperations {seed=[35F19948C8D0B296:4C8BC51BAD80E140]}

Error Message:
[Intersects] qIdx:46 Shouldn't match I#1:org.apache.lucene.spatial.spatial4j.Geo3dShape@17ea321b Q:org.apache.lucene.spatial.spatial4j.Geo3dShape@49405867

Stack Trace:
java.lang.AssertionError: [Intersects] qIdx:46 Shouldn't match I#1:org.apache.lucene.spatial.spatial4j.Geo3dShape@17ea321b Q:org.apache.lucene.spatial.spatial4j.Geo3dShape@49405867
	at __randomizedtesting.SeedInfo.seed([35F19948C8D0B296:4C8BC51BAD80E140]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:55)
	at org.apache.lucene.spatial.spatial4j.Geo3dRptTest.testOperations(Geo3dRptTest.java:117)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.LeaderElectionContextKeyTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([D426A0545BC5456:851655DFEB4039AE]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.solr.cloud.LeaderElectionContextKeyTest.test(LeaderElectionContextKeyTest.java:88)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 9348 lines...]
   [junit4] Suite: org.apache.lucene.spatial.spatial4j.Geo3dRptTest
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=Geo3dRptTest -Dtests.method=testOperations -Dtests.seed=35F19948C8D0B296 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-MT -Dtests.timezone=Africa/Porto-Novo -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 3.13s J2 | Geo3dRptTest.testOperations {seed=[35F19948C8D0B296:4C8BC51BAD80E140]} <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: [Intersects] qIdx:46 Shouldn't match I#1:org.apache.lucene.spatial.spatial4j.Geo3dShape@17ea321b Q:org.apache.lucene.spatial.spatial4j.Geo3dShape@49405867
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([35F19948C8D0B296:4C8BC51BAD80E140]:0)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
   [junit4]    > 	at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:55)
   [junit4]    > 	at org.apache.lucene.spatial.spatial4j.Geo3dRptTest.testOperations(Geo3dRptTest.java:117)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {Geo3dRptTest_rpt=PostingsFormat(name=Memory), id=PostingsFormat(name=Memory)}, docValues:{Geo3dRptTest_sdv=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=478, maxMBSortInHeap=5.983076427221281, sim=RandomSimilarity(queryNorm=false): {}, locale=en-MT, timezone=Africa/Porto-Novo
   [junit4]   2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=1,free=10556296,total=25817088
   [junit4]   2> NOTE: All tests run in this JVM: [Geo3dRptTest]
   [junit4] Completed [17/23 (1!)] on J2 in 6.97s, 13 tests, 1 failure <<< FAILURES!

[...truncated 2808 lines...]
   [junit4] Suite: org.apache.solr.cloud.LeaderElectionContextKeyTest
   [junit4]   2> 713289 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/init-core-data-001
   [junit4]   2> 713290 WARN  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=9 numCloses=9
   [junit4]   2> 713290 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 713291 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 713293 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001
   [junit4]   2> 713294 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 713294 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 713294 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 713296 ERROR (Thread-1606) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 713394 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:32955
   [junit4]   2> 713404 INFO  (jetty-launcher-994-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 713405 INFO  (jetty-launcher-994-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@38e81f{/solr,null,AVAILABLE}
   [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@d0d870{SSL,[ssl, http/1.1]}{127.0.0.1:33501}
   [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ] o.e.j.s.Server Started @714536ms
   [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33501}
   [junit4]   2> 713406 ERROR (jetty-launcher-994-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-07T22:56:28.682Z
   [junit4]   2> 713411 INFO  (jetty-launcher-994-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 713422 INFO  (jetty-launcher-994-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32955/solr
   [junit4]   2> 713481 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 713481 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33501_solr
   [junit4]   2> 713482 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.c.Overseer Overseer (id=98790079762989059-127.0.0.1:33501_solr-n_0000000000) starting
   [junit4]   2> 713487 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33501_solr
   [junit4]   2> 713490 INFO  (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 713640 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
   [junit4]   2> 713663 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
   [junit4]   2> 713663 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
   [junit4]   2> 713665 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/.
   [junit4]   2> 713693 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 713694 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:32955/solr ready
   [junit4]   2> 713797 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 713802 INFO  (OverseerThreadFactory-2413-thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.CreateCollectionCmd Create collection testCollection1
   [junit4]   2> 713908 INFO  (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testCollection1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testCollection1_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
   [junit4]   2>   "type":"NRT"} 
   [junit4]   2> 713912 INFO  (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testCollection1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testCollection1_shard2_replica_n3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
   [junit4]   2>   "type":"NRT"} 
   [junit4]   2> 714119 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=config&newCollection=true&name=testCollection1_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 714119 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 714119 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection1_shard2_replica_n3&action=CREATE&numShards=2&collection=testCollection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 714227 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 714227 INFO  (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 715133 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 715134 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 715141 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.s.IndexSchema [testCollection1_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 715141 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.s.IndexSchema [testCollection1_shard2_replica_n3] Schema name=minimal
   [junit4]   2> 715158 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 715159 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.CoreContainer Creating SolrCore 'testCollection1_shard2_replica_n3' using configuration from collection testCollection1, trusted=true
   [junit4]   2> 715159 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 715159 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testCollection1_shard1_replica_n1' using configuration from collection testCollection1, trusted=true
   [junit4]   2> 715160 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.core.testCollection1.shard1.replica_n1' (registry 'solr.core.testCollection1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
   [junit4]   2> 715160 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.core.testCollection1.shard2.replica_n3' (registry 'solr.core.testCollection1.shard2.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
   [junit4]   2> 715160 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 715161 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 715161 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore [[testCollection1_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection1_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection1_shard1_replica_n1/data/]
   [junit4]   2> 715161 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore [[testCollection1_shard2_replica_n3] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection1_shard2_replica_n3], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection1_shard2_replica_n3/data/]
   [junit4]   2> 715228 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 715228 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 715230 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 715230 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 715233 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.s.SolrIndexSearcher Opening [Searcher@15532e2[testCollection1_shard2_replica_n3] main]
   [junit4]   2> 715238 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 715239 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 715239 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 715240 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 715241 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 715241 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 715241 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 715241 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1580641278248353792
   [junit4]   2> 715243 INFO  (searcherExecutor-2416-thread-1-processing-n:127.0.0.1:33501_solr x:testCollection1_shard2_replica_n3 s:shard2 c:testCollection1 r:core_node4) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore [testCollection1_shard2_replica_n3] Registered new searcher Searcher@15532e2[testCollection1_shard2_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 715244 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1202a80[testCollection1_shard1_replica_n1] main]
   [junit4]   2> 715246 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 715246 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 715248 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 715249 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 715249 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 715250 INFO  (searcherExecutor-2417-thread-1-processing-n:127.0.0.1:33501_solr x:testCollection1_shard1_replica_n1 s:shard1 c:testCollection1 r:core_node2) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore [testCollection1_shard1_replica_n1] Registered new searcher Searcher@1202a80[testCollection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33501/solr/testCollection1_shard2_replica_n3/
   [junit4]   2> 715250 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1580641278257790976
   [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy https://127.0.0.1:33501/solr/testCollection1_shard2_replica_n3/ has no replicas
   [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 715253 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:33501/solr/testCollection1_shard2_replica_n3/ shard2
   [junit4]   2> 715256 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 715256 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33501/solr/testCollection1_shard1_replica_n1/
   [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:33501/solr/testCollection1_shard1_replica_n1/ has no replicas
   [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 715260 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:33501/solr/testCollection1_shard1_replica_n1/ shard1
   [junit4]   2> 715363 INFO  (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 715363 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 715405 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 715406 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection1_shard2_replica_n3&action=CREATE&numShards=2&collection=testCollection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1287
   [junit4]   2> 715412 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 715415 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=config&newCollection=true&name=testCollection1_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1296
   [junit4]   2> 715419 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 715517 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 715517 INFO  (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 715803 INFO  (OverseerCollectionConfigSetProcessor-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 716419 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2622
   [junit4]   2> 716421 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection2&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 716422 INFO  (OverseerThreadFactory-2413-thread-2-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.CreateCollectionCmd Create collection testCollection2
   [junit4]   2> 716627 INFO  (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testCollection2",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testCollection2_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
   [junit4]   2>   "type":"NRT"} 
   [junit4]   2> 716629 INFO  (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testCollection2",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testCollection2_shard2_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
   [junit4]   2>   "type":"NRT"} 
   [junit4]   2> 716831 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection2_shard2_replica_n2&action=CREATE&numShards=2&collection=testCollection2&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 716833 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=testCollection2_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection2&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 716937 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 716937 INFO  (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 717844 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 717844 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 717870 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.s.IndexSchema [testCollection2_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 717870 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.s.IndexSchema [testCollection2_shard2_replica_n2] Schema name=minimal
   [junit4]   2> 717874 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 717874 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 717874 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testCollection2_shard1_replica_n1' using configuration from collection testCollection2, trusted=true
   [junit4]   2> 717874 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testCollection2_shard2_replica_n2' using configuration from collection testCollection2, trusted=true
   [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.core.testCollection2.shard2.replica_n2' (registry 'solr.core.testCollection2.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
   [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.core.testCollection2.shard1.replica_n1' (registry 'solr.core.testCollection2.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
   [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore [[testCollection2_shard2_replica_n2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection2_shard2_replica_n2], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection2_shard2_replica_n2/data/]
   [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore [[testCollection2_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection2_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection2_shard1_replica_n1/data/]
   [junit4]   2> 717939 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 717940 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 717941 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 717941 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 717942 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 717942 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7b3d53[testCollection2_shard1_replica_n1] main]
   [junit4]   2> 717942 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 717943 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 717944 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 717944 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 717944 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 717945 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 717946 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@17e3347[testCollection2_shard2_replica_n2] main]
   [junit4]   2> 717946 INFO  (searcherExecutor-2426-thread-1-processing-n:127.0.0.1:33501_solr x:testCollection2_shard1_replica_n1 s:shard1 c:testCollection2 r:core_node3) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore [testCollection2_shard1_replica_n1] Registered new searcher Searcher@7b3d53[testCollection2_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 717946 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1580641281084751872
   [junit4]   2> 717946 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 717947 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 717947 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 717948 INFO  (searcherExecutor-2427-thread-1-processing-n:127.0.0.1:33501_solr x:testCollection2_shard2_replica_n2 s:shard2 c:testCollection2 r:core_node4) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore [testCollection2_shard2_replica_n2] Registered new searcher Searcher@17e3347[testCollection2_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 717949 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1580641281087897600
   [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33501/solr/testCollection2_shard1_replica_n1/
   [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:33501/solr/testCollection2_shard1_replica_n1/ has no replicas
   [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33501/solr/testCollection2_shard2_replica_n2/
   [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy https://127.0.0.1:33501/solr/testCollection2_shard2_replica_n2/ has no replicas
   [junit4]   2> 717976 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 717977 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:33501/solr/testCollection2_shard1_replica_n1/ shard1
   [junit4]   2> 717978 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:33501/solr/testCollection2_shard2_replica_n2/ shard2
   [junit4]   2> 718081 INFO  (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 718082 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 718129 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 718129 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 718130 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=testCollection2_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection2&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1296
   [junit4]   2> 718130 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection2_shard2_replica_n2&action=CREATE&numShards=2&collection=testCollection2&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1299
   [junit4]   2> 718133 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 718234 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 718234 INFO  (zkCallback-1001-thread-3-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 718423 INFO  (OverseerCollectionConfigSetProcessor-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 719133 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection2&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2712
   [junit4]   2> 719134 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: testCollection1 failOnTimeout:true timeout (sec):30
   [junit4]   2> 719134 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: testCollection1
   [junit4]   2> 719134 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: testCollection2 failOnTimeout:true timeout (sec):30
   [junit4]   2> 719134 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: testCollection2
   [junit4]   2> 719145 INFO  (TEST-LeaderElectionContextKeyTest.test-seed#[D426A0545BC5456]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 719153 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection1.shard2.replica_n3, tag=null
   [junit4]   2> 719154 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@b5e43b: rootName = solr_33501, domain = solr.core.testCollection1.shard2.replica_n3, service url = null, agent id = null] for registry solr.core.testCollection1.shard2.replica_n3 / com.codahale.metrics.MetricRegistry@100889b
   [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.c.SolrCore [testCollection1_shard2_replica_n3]  CLOSING SolrCore org.apache.solr.core.SolrCore@8e18e5
   [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection1.shard2.replica_n3, tag=9312485
   [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCollection1.shard2.leader, tag=9312485
   [junit4]   2> 719179 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=false&core=testCollection1_shard2_replica_n3&deleteDataDir=false&action=UNLOAD&indexInfo=true&wt=javabin&version=2&deleteIndex=false} status=0 QTime=26
   [junit4]   2> 719181 INFO  (TEST-LeaderElectionContextKeyTest.test-seed#[D426A0545BC5456]) [    ] o.a.s.SolrTestCaseJ4 ###Ending test
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=LeaderElectionContextKeyTest -Dtests.method=test -Dtests.seed=D426A0545BC5456 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mk -Dtests.timezone=Canada/Mountain -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.05s J1 | LeaderElectionContextKeyTest.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D426A0545BC5456:851655DFEB4039AE]:0)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElectionContextKeyTest.test(LeaderElectionContextKeyTest.java:88)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 719191 INFO  (jetty-closer-995-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@d0d870{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=31261178
   [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@cfcf8d: rootName = solr_33501, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@8cc4d4
   [junit4]   2> 719200 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 719200 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@18f30a8: rootName = solr_33501, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@1bf26ed
   [junit4]   2> 719208 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 719208 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@f110c8: rootName = solr_33501, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@192a8b1
   [junit4]   2> 719209 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-1) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore [testCollection1_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@16aac00
   [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-1) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection1.shard1.replica_n1, tag=23768064
   [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-2) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore [testCollection2_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@17509de
   [junit4]   2> 719215 INFO  (coreCloseExecutor-2436-thread-3) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore [testCollection2_shard2_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@140f2a1
   [junit4]   2> 719216 INFO  (coreCloseExecutor-2436-thread-1) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@169f449: rootName = solr_33501, domain = solr.core.testCollection1.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.testCollection1.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@121b656
   [junit4]   2> 719230 INFO  (coreCloseExecutor-2436-thread-2) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection2.shard1.replica_n1, tag=24447454
   [junit4]   2> 719230 INFO  (coreCloseExecutor-2436-thread-2) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@e8b978: rootName = solr_33501, domain = solr.core.testCollection2.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.testCollection2.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@113466a
   [junit4]   2> 719244 INFO  (coreCloseExecutor-2436-thread-3) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection2.shard2.replica_n2, tag=21033633
   [junit4]   2> 719244 INFO  (coreCloseExecutor-2436-thread-3) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@cfc71: rootName = solr_33501, domain = solr.core.testCollection2.shard2.replica_n2, service url = null, agent id = null] for registry solr.core.testCollection2.shard2.replica_n2 / com.codahale.metrics.MetricRegistry@13e256e
   [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-1) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCollection1.shard1.leader, tag=23768064
   [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-2) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCollection2.shard1.leader, tag=24447454
   [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-3) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCollection2.shard2.leader, tag=21033633
   [junit4]   2> 719267 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.c.Overseer Overseer (id=98790079762989059-127.0.0.1:33501_solr-n_0000000000) closing
   [junit4]   2> 719267 INFO  (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:33501_solr
   [junit4]   2> 720768 WARN  (zkCallback-1001-thread-3-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 720768 INFO  (jetty-closer-995-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@38e81f{/solr,null,UNAVAILABLE}
   [junit4]   2> 720770 ERROR (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 720770 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:32955 32955
   [junit4]   2> 725792 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:32955 32955
   [junit4]   2> 725792 WARN  (Thread-1606) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/aliases.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/collections/testCollection2/state.json
   [junit4]   2> 	5	/solr/collections/testCollection1/state.json
   [junit4]   2> 	2	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/clusterprops.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/live_nodes
   [junit4]   2> 	2	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1086, maxMBSortInHeap=5.302125034778907, sim=RandomSimilarity(queryNorm=true): {}, locale=mk, timezone=Canada/Mountain
   [junit4]   2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=1,free=134427200,total=326377472
   [junit4]   2> NOTE: All tests run in this JVM: [BasicFunctionalityTest, TestTestInjection, TestConfigSetsAPIZkFailure, TestNRTOpen, TestPolicyCloud, CollectionsAPIAsyncDistributedZkTest, DistributedDebugComponentTest, TestFieldTypeCollectionResource, SearchHandlerTest, TestHighFrequencyDictionaryFactory, DistributedIntervalFacetingTest, TestReRankQParserPlugin, DistribJoinFromCollectionTest, TestRTGBase, BigEndianAscendingWordDeserializerTest, DocValuesMultiTest, SolrCloudExampleTest, TestDistribDocBasedVersion, CoreAdminHandlerTest, TestCloudSchemaless, TestManagedSynonymFilterFactory, TestEmbeddedSolrServerAdminHandler, TestDistributedGrouping, BufferStoreTest, TestMissingGroups, TestPushWriter, PrimUtilsTest, TestNumericTerms64, ShardRoutingCustomTest, TestSimpleTrackingShardHandler, AddSchemaFieldsUpdateProcessorFactoryTest, TestCollectionsAPIViaSolrCloudCluster, TestCloudSearcherWarming, TestHashPartitioner, ClassificationUpdateProcessorFactoryTest, TestSolrIndexConfig, RulesTest, JsonLoaderTest, SOLR749Test, TestHdfsBackupRestoreCore, BadComponentTest, TestBulkSchemaAPI, BooleanFieldTest, TestSkipOverseerOperations, AsyncCallRequestStatusResponseTest, TestUseDocValuesAsStored, TestConfigSetImmutable, TestGeoJSONResponseWriter, TestCollectionAPIs, TestPKIAuthenticationPlugin, UUIDUpdateProcessorFallbackTest, SystemInfoHandlerTest, ParsingFieldUpdateProcessorsTest, ExplicitHLLTest, TestBlendedInfixSuggestions, TestCloudNestedDocsSort, ZkCLITest, UninvertDocValuesMergePolicyTest, TestRandomRequestDistribution, LeaderElectionContextKeyTest]
   [junit4] Completed [189/734 (1!)] on J1 in 12.52s, 1 test, 1 failure <<< FAILURES!

[...truncated 44564 lines...]

Re: Geo3dRptTest.testOperations failure

Posted by Karl Wright <da...@gmail.com>.
I've assigned the ticket to myself, and will have a look, but I can't look
until later in the week I'm afraid...

Karl

On Mon, Oct 9, 2017 at 10:57 PM, David Smiley <da...@gmail.com>
wrote:

> I added toString a few minutes ago (making it to 7.1 release branch).  I
> posted reproducible test failure here: https://issues.apache.
> org/jira/browse/LUCENE-7990
>
> If you can't get to the problem, so be it.  There's a decent chance this
> is just some test error tolerance/precision issue.
>
> Have a nice holiday/vacation.
>
> ~ David
>
>
> On Sun, Oct 8, 2017 at 11:53 AM Ignacio Vera Sequeiros <iv...@eso.org>
> wrote:
>
>>
>> Hi David,
>>
>> We should implement toString method indeed. It can just call the toString
>> method of the Wrapped shape.
>>
>> I suspect the error might come from a new shape added to support exact
>> circles. If true, I am very interested to understand why is failing.
>>
>> I am on holidays for the next two weeks and with limited internet
>> connection. So if it can wait, I will have a look when I am back.
>>
>>
>> ------------------------------
>> *From:* David Smiley <da...@gmail.com>
>> *Sent:* Sunday, October 8, 2017 4:19:34 AM
>> *To:* dev@lucene.apache.org; Ignacio Vera Sequeiros
>> *Subject:* Geo3dRptTest.testOperations failure
>>
>> I'd file a JIRA issue but it's down at the moment.  Ignacio: any idea
>> about this one?
>>
>> I can see that there is no Geo3dShape.toString; we ought to implement
>> that.
>>
>> On Sat, Oct 7, 2017 at 7:51 PM Policeman Jenkins Server <
>> jenkins@thetaphi.de> wrote:
>>
>>> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/566/
>>> Java: 32bit/jdk1.8.0_144 -client -XX:+UseSerialGC
>>>
>>> 2 tests failed.
>>> FAILED:  org.apache.lucene.spatial.spatial4j.Geo3dRptTest.testOperations
>>> {seed=[35F19948C8D0B296:4C8BC51BAD80E140]}
>>>
>>> Error Message:
>>> [Intersects] qIdx:46 Shouldn't match I#1:org.apache.lucene.spatial.
>>> spatial4j.Geo3dShape@17ea321b Q:org.apache.lucene.spatial.
>>> spatial4j.Geo3dShape@49405867
>>>
>>> Stack Trace:
>>> java.lang.AssertionError: [Intersects] qIdx:46 Shouldn't match
>>> I#1:org.apache.lucene.spatial.spatial4j.Geo3dShape@17ea321b
>>> Q:org.apache.lucene.spatial.spatial4j.Geo3dShape@49405867
>>>         at __randomizedtesting.SeedInfo.seed([35F19948C8D0B296:
>>> 4C8BC51BAD80E140]:0)
>>>         at org.junit.Assert.fail(Assert.java:93)
>>>         at org.apache.lucene.spatial.prefix.
>>> RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCas
>>> e.java:126)
>>>         at org.apache.lucene.spatial.prefix.
>>> RandomSpatialOpStrategyTestCase.testOperation(
>>> RandomSpatialOpStrategyTestCase.java:115)
>>>         at org.apache.lucene.spatial.prefix.
>>> RandomSpatialOpStrategyTestCase.testOperationRandomShapes(
>>> RandomSpatialOpStrategyTestCase.java:55)
>>>         at org.apache.lucene.spatial.spatial4j.Geo3dRptTest.
>>> testOperations(Geo3dRptTest.java:117)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke(
>>> NativeMethodAccessorImpl.java:62)
>>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>> DelegatingMethodAccessorImpl.java:43)
>>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(
>>> RandomizedRunner.java:1737)
>>>         at com.carrotsearch.randomizedtesting.
>>> RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>>>         at com.carrotsearch.randomizedtesting.
>>> RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>>>         at com.carrotsearch.randomizedtesting.
>>> RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>>>         at org.apache.lucene.util.TestRuleSetupTeardownChained$
>>> 1.evaluate(TestRuleSetupTeardownChained.java:49)
>>>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(
>>> AbstractBeforeAfterRule.java:45)
>>>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(
>>> TestRuleThreadAndTestName.java:48)
>>>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures
>>> $1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>>         at org.apache.lucene.util.TestRuleMarkFailure$1.
>>> evaluate(TestRuleMarkFailure.java:47)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$
>>> StatementRunner.run(ThreadLeakControl.java:368)
>>>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.
>>> forkTimeoutingTask(ThreadLeakControl.java:817)
>>>         at com.carrotsearch.randomizedtesting.
>>> ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>>>         at com.carrotsearch.randomizedtesting.RandomizedRunner.
>>> runSingleTest(RandomizedRunner.java:943)
>>>         at com.carrotsearch.randomizedtesting.
>>> RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>>>         at com.carrotsearch.randomizedtesting.
>>> RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>>>         at com.carrotsearch.randomizedtesting.
>>> RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>>>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(
>>> AbstractBeforeAfterRule.java:45)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(
>>> TestRuleStoreClassName.java:41)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> NoShadowingOrOverridesOnMethodsRule$1.evaluate(
>>> NoShadowingOrOverridesOnMethodsRule.java:40)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> NoShadowingOrOverridesOnMethodsRule$1.evaluate(
>>> NoShadowingOrOverridesOnMethodsRule.java:40)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(
>>> TestRuleAssertionsRequired.java:53)
>>>         at org.apache.lucene.util.TestRuleMarkFailure$1.
>>> evaluate(TestRuleMarkFailure.java:47)
>>>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures
>>> $1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(
>>> TestRuleIgnoreTestSuites.java:54)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$
>>> StatementRunner.run(ThreadLeakControl.java:368)
>>>         at java.lang.Thread.run(Thread.java:748)
>>>
>>>
>>> FAILED:  org.apache.solr.cloud.LeaderElectionContextKeyTest.test
>>>
>>> Error Message:
>>>
>>>
>>> Stack Trace:
>>> java.lang.AssertionError
>>>         at __randomizedtesting.SeedInfo.seed([D426A0545BC5456:
>>> 851655DFEB4039AE]:0)
>>>         at org.junit.Assert.fail(Assert.java:92)
>>>         at org.junit.Assert.assertTrue(Assert.java:43)
>>>         at org.junit.Assert.assertTrue(Assert.java:54)
>>>         at org.apache.solr.cloud.LeaderElectionContextKeyTest.test(
>>> LeaderElectionContextKeyTest.java:88)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke(
>>> NativeMethodAccessorImpl.java:62)
>>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>> DelegatingMethodAccessorImpl.java:43)
>>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(
>>> RandomizedRunner.java:1737)
>>>         at com.carrotsearch.randomizedtesting.
>>> RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>>>         at com.carrotsearch.randomizedtesting.
>>> RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>>>         at com.carrotsearch.randomizedtesting.
>>> RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.
>>> java:57)
>>>         at org.apache.lucene.util.TestRuleSetupTeardownChained$
>>> 1.evaluate(TestRuleSetupTeardownChained.java:49)
>>>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(
>>> AbstractBeforeAfterRule.java:45)
>>>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(
>>> TestRuleThreadAndTestName.java:48)
>>>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures
>>> $1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>>         at org.apache.lucene.util.TestRuleMarkFailure$1.
>>> evaluate(TestRuleMarkFailure.java:47)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$
>>> StatementRunner.run(ThreadLeakControl.java:368)
>>>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.
>>> forkTimeoutingTask(ThreadLeakControl.java:817)
>>>         at com.carrotsearch.randomizedtesting.
>>> ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>>>         at com.carrotsearch.randomizedtesting.RandomizedRunner.
>>> runSingleTest(RandomizedRunner.java:943)
>>>         at com.carrotsearch.randomizedtesting.
>>> RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>>>         at com.carrotsearch.randomizedtesting.
>>> RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>>>         at com.carrotsearch.randomizedtesting.
>>> RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.
>>> java:57)
>>>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(
>>> AbstractBeforeAfterRule.java:45)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(
>>> TestRuleStoreClassName.java:41)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> NoShadowingOrOverridesOnMethodsRule$1.evaluate(
>>> NoShadowingOrOverridesOnMethodsRule.java:40)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> NoShadowingOrOverridesOnMethodsRule$1.evaluate(
>>> NoShadowingOrOverridesOnMethodsRule.java:40)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(
>>> TestRuleAssertionsRequired.java:53)
>>>         at org.apache.lucene.util.TestRuleMarkFailure$1.
>>> evaluate(TestRuleMarkFailure.java:47)
>>>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures
>>> $1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(
>>> TestRuleIgnoreTestSuites.java:54)
>>>         at com.carrotsearch.randomizedtesting.rules.
>>> StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$
>>> StatementRunner.run(ThreadLeakControl.java:368)
>>>         at java.lang.Thread.run(Thread.java:748)
>>>
>>>
>>>
>>>
>>> Build Log:
>>> [...truncated 9348 lines...]
>>>    [junit4] Suite: org.apache.lucene.spatial.spatial4j.Geo3dRptTest
>>>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=Geo3dRptTest
>>> -Dtests.method=testOperations -Dtests.seed=35F19948C8D0B296
>>> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-MT
>>> -Dtests.timezone=Africa/Porto-Novo -Dtests.asserts=true
>>> -Dtests.file.encoding=UTF-8
>>>    [junit4] FAILURE 3.13s J2 | Geo3dRptTest.testOperations
>>> {seed=[35F19948C8D0B296:4C8BC51BAD80E140]} <<<
>>>    [junit4]    > Throwable #1: java.lang.AssertionError: [Intersects]
>>> qIdx:46 Shouldn't match I#1:org.apache.lucene.spatial.
>>> spatial4j.Geo3dShape@17ea321b Q:org.apache.lucene.spatial.
>>> spatial4j.Geo3dShape@49405867
>>>    [junit4]    >        at __randomizedtesting.SeedInfo.
>>> seed([35F19948C8D0B296:4C8BC51BAD80E140]:0)
>>>    [junit4]    >        at org.apache.lucene.spatial.prefix.
>>> RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCas
>>> e.java:126)
>>>    [junit4]    >        at org.apache.lucene.spatial.prefix.
>>> RandomSpatialOpStrategyTestCase.testOperation(
>>> RandomSpatialOpStrategyTestCase.java:115)
>>>    [junit4]    >        at org.apache.lucene.spatial.prefix.
>>> RandomSpatialOpStrategyTestCase.testOperationRandomShapes(
>>> RandomSpatialOpStrategyTestCase.java:55)
>>>    [junit4]    >        at org.apache.lucene.spatial.
>>> spatial4j.Geo3dRptTest.testOperations(Geo3dRptTest.java:117)
>>>    [junit4]    >        at java.lang.Thread.run(Thread.java:748)
>>>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70):
>>> {Geo3dRptTest_rpt=PostingsFormat(name=Memory),
>>> id=PostingsFormat(name=Memory)}, docValues:{Geo3dRptTest_sdv=
>>> DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=478,
>>> maxMBSortInHeap=5.983076427221281, sim=RandomSimilarity(queryNorm=false):
>>> {}, locale=en-MT, timezone=Africa/Porto-Novo
>>>    [junit4]   2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation
>>> 1.8.0_144 (32-bit)/cpus=8,threads=1,free=10556296,total=25817088
>>>    [junit4]   2> NOTE: All tests run in this JVM: [Geo3dRptTest]
>>>    [junit4] Completed [17/23 (1!)] on J2 in 6.97s, 13 tests, 1 failure
>>> <<< FAILURES!
>>>
>>> [...truncated 2808 lines...]
>>>    [junit4] Suite: org.apache.solr.cloud.LeaderElectionContextKeyTest
>>>    [junit4]   2> 713289 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null
>>> & java.security.egd=file:/dev/./urandom
>>>    [junit4]   2> Creating dataDir: /home/jenkins/workspace/
>>> Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.
>>> LeaderElectionContextKeyTest_D426A0545BC5456-001/init-core-data-001
>>>    [junit4]   2> 713290 WARN  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=9 numCloses=9
>>>    [junit4]   2> 713290 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true)
>>> w/NUMERIC_DOCVALUES_SYSPROP=false
>>>    [junit4]   2> 713291 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true)
>>> via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN,
>>> clientAuth=NaN)
>>>    [junit4]   2> 713293 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in
>>> /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/
>>> build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_
>>> D426A0545BC5456-001/tempDir-001
>>>    [junit4]   2> 713294 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
>>>    [junit4]   2> 713294 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer
>>> client port:0.0.0.0/0.0.0.0:0
>>>    [junit4]   2> 713294 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer
>>> Starting server
>>>    [junit4]   2> 713296 ERROR (Thread-1606) [    ]
>>> o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper
>>> server won't take any action on ERROR or SHUTDOWN server state changes
>>>    [junit4]   2> 713394 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.s.c.ZkTestServer start zk server on port:32955
>>>    [junit4]   2> 713404 INFO  (jetty-launcher-994-thread-1) [    ]
>>> o.e.j.s.Server jetty-9.3.20.v20170531
>>>    [junit4]   2> 713405 INFO  (jetty-launcher-994-thread-1) [    ]
>>> o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@
>>> 38e81f{/solr,null,AVAILABLE}
>>>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
>>> o.e.j.s.AbstractConnector Started ServerConnector@d0d870{SSL,[ssl,
>>> http/1.1]}{127.0.0.1:33501}
>>>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
>>> o.e.j.s.Server Started @714536ms
>>>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
>>> o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr,
>>> hostPort=33501}
>>>    [junit4]   2> 713406 ERROR (jetty-launcher-994-thread-1) [    ]
>>> o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may
>>> be missing or incomplete.
>>>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
>>> o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr?
>>> version 7.1.0
>>>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
>>> o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port
>>> null
>>>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
>>> o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default
>>> config dir: null
>>>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
>>> o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time:
>>> 2017-10-07T22:56:28.682Z
>>>    [junit4]   2> 713411 INFO  (jetty-launcher-994-thread-1) [    ]
>>> o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>>>    [junit4]   2> 713422 INFO  (jetty-launcher-994-thread-1) [    ]
>>> o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32955/solr
>>>    [junit4]   2> 713481 INFO  (jetty-launcher-994-thread-1)
>>> [n:127.0.0.1:33501_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
>>>    [junit4]   2> 713481 INFO  (jetty-launcher-994-thread-1)
>>> [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerElectionContext I am going
>>> to be the leader 127.0.0.1:33501_solr
>>>    [junit4]   2> 713482 INFO  (jetty-launcher-994-thread-1)
>>> [n:127.0.0.1:33501_solr    ] o.a.s.c.Overseer Overseer
>>> (id=98790079762989059-127.0.0.1:33501_solr-n_0000000000) starting
>>>    [junit4]   2> 713487 INFO  (jetty-launcher-994-thread-1)
>>> [n:127.0.0.1:33501_solr    ] o.a.s.c.ZkController Register node as live
>>> in ZooKeeper:/live_nodes/127.0.0.1:33501_solr
>>>    [junit4]   2> 713490 INFO  (OverseerStateUpdate-
>>> 98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
>>>    [junit4]   2> 713640 INFO  (jetty-launcher-994-thread-1)
>>> [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring
>>> for 'solr_33501.solr.node' (registry 'solr.node') enabled at server:
>>> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>>>    [junit4]   2> 713663 INFO  (jetty-launcher-994-thread-1)
>>> [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring
>>> for 'solr_33501.solr.jvm' (registry 'solr.jvm') enabled at server:
>>> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>>>    [junit4]   2> 713663 INFO  (jetty-launcher-994-thread-1)
>>> [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring
>>> for 'solr_33501.solr.jetty' (registry 'solr.jetty') enabled at server:
>>> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>>>    [junit4]   2> 713665 INFO  (jetty-launcher-994-thread-1)
>>> [n:127.0.0.1:33501_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core
>>> definitions underneath /home/jenkins/workspace/
>>> Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.
>>> LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/.
>>>    [junit4]   2> 713693 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) ->
>>> (1)
>>>    [junit4]   2> 713694 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at
>>> 127.0.0.1:32955/solr ready
>>>    [junit4]   2> 713797 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with
>>> params replicationFactor=1&collection.configName=config&
>>> maxShardsPerNode=1000&name=testCollection1&nrtReplicas=1&
>>> action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
>>>    [junit4]   2> 713802 INFO  (OverseerThreadFactory-2413-
>>> thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr
>>> ] o.a.s.c.CreateCollectionCmd Create collection testCollection1
>>>    [junit4]   2> 713908 INFO  (OverseerStateUpdate-
>>> 98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.c.o.SliceMutator createReplica() {
>>>    [junit4]   2>   "operation":"ADDREPLICA",
>>>    [junit4]   2>   "collection":"testCollection1",
>>>    [junit4]   2>   "shard":"shard1",
>>>    [junit4]   2>   "core":"testCollection1_shard1_replica_n1",
>>>    [junit4]   2>   "state":"down",
>>>    [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
>>>    [junit4]   2>   "type":"NRT"}
>>>    [junit4]   2> 713912 INFO  (OverseerStateUpdate-
>>> 98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.c.o.SliceMutator createReplica() {
>>>    [junit4]   2>   "operation":"ADDREPLICA",
>>>    [junit4]   2>   "collection":"testCollection1",
>>>    [junit4]   2>   "shard":"shard2",
>>>    [junit4]   2>   "core":"testCollection1_shard2_replica_n3",
>>>    [junit4]   2>   "state":"down",
>>>    [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
>>>    [junit4]   2>   "type":"NRT"}
>>>    [junit4]   2> 714119 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.h.a.CoreAdminOperation core create command
>>> qt=/admin/cores&coreNodeName=core_node2&collection.configName=config&
>>> newCollection=true&name=testCollection1_shard1_replica_n1&action=CREATE&
>>> numShards=2&collection=testCollection1&shard=shard1&
>>> wt=javabin&version=2&replicaType=NRT
>>>    [junit4]   2> 714119 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache
>>> for 2147483647 <(214)%20748-3647> transient cores
>>>    [junit4]   2> 714119 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.h.a.CoreAdminOperation core create command
>>> qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&
>>> newCollection=true&name=testCollection1_shard2_replica_n3&action=CREATE&
>>> numShards=2&collection=testCollection1&shard=shard2&
>>> wt=javabin&version=2&replicaType=NRT
>>>    [junit4]   2> 714227 INFO  (zkCallback-1001-thread-1-
>>> processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ]
>>> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
>>> state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json]
>>> for collection [testCollection1] has occurred - updating... (live nodes
>>> size: [1])
>>>    [junit4]   2> 714227 INFO  (zkCallback-1001-thread-2-
>>> processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ]
>>> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
>>> state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json]
>>> for collection [testCollection1] has occurred - updating... (live nodes
>>> size: [1])
>>>    [junit4]   2> 715133 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
>>>    [junit4]   2> 715134 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
>>>    [junit4]   2> 715141 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.s.IndexSchema [testCollection1_shard1_replica_n1] Schema
>>> name=minimal
>>>    [junit4]   2> 715141 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.s.IndexSchema [testCollection1_shard2_replica_n3] Schema
>>> name=minimal
>>>    [junit4]   2> 715158 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
>>>    [junit4]   2> 715159 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.c.CoreContainer Creating SolrCore 'testCollection1_shard2_replica_n3'
>>> using configuration from collection testCollection1, trusted=true
>>>    [junit4]   2> 715159 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
>>>    [junit4]   2> 715159 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.c.CoreContainer Creating SolrCore 'testCollection1_shard1_replica_n1'
>>> using configuration from collection testCollection1, trusted=true
>>>    [junit4]   2> 715160 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.core.
>>> testCollection1.shard1.replica_n1' (registry 'solr.core.testCollection1.shard1.replica_n1')
>>> enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>>>    [junit4]   2> 715160 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.core.
>>> testCollection1.shard2.replica_n3' (registry 'solr.core.testCollection1.shard2.replica_n3')
>>> enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>>>    [junit4]   2> 715160 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
>>>    [junit4]   2> 715161 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
>>>    [junit4]   2> 715161 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.c.SolrCore [[testCollection1_shard1_replica_n1] ] Opening new
>>> SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/
>>> build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_
>>> D426A0545BC5456-001/tempDir-001/node1/testCollection1_shard1_replica_n1],
>>> dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-
>>> Linux/solr/build/solr-core/test/J1/temp/solr.cloud.
>>> LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-
>>> 001/node1/./testCollection1_shard1_replica_n1/data/]
>>>    [junit4]   2> 715161 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.c.SolrCore [[testCollection1_shard2_replica_n3] ] Opening new
>>> SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/
>>> build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_
>>> D426A0545BC5456-001/tempDir-001/node1/testCollection1_shard2_replica_n3],
>>> dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-
>>> Linux/solr/build/solr-core/test/J1/temp/solr.cloud.
>>> LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-
>>> 001/node1/./testCollection1_shard2_replica_n3/data/]
>>>    [junit4]   2> 715228 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.u.UpdateHandler Using UpdateLog implementation:
>>> org.apache.solr.update.UpdateLog
>>>    [junit4]   2> 715228 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null
>>> defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
>>> numVersionBuckets=65536
>>>    [junit4]   2> 715230 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.u.CommitTracker Hard AutoCommit: disabled
>>>    [junit4]   2> 715230 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.u.CommitTracker Soft AutoCommit: disabled
>>>    [junit4]   2> 715233 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.s.SolrIndexSearcher Opening [Searcher@15532e2[
>>> testCollection1_shard2_replica_n3] main]
>>>    [junit4]   2> 715238 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with
>>> znodeBase: /configs/config
>>>    [junit4]   2> 715239 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.u.UpdateHandler Using UpdateLog implementation:
>>> org.apache.solr.update.UpdateLog
>>>    [junit4]   2> 715239 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null
>>> defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
>>> numVersionBuckets=65536
>>>    [junit4]   2> 715240 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
>>> ZooKeeperStorageIO:path=/configs/config
>>>    [junit4]   2> 715241 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.u.CommitTracker Hard AutoCommit: disabled
>>>    [junit4]   2> 715241 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
>>>    [junit4]   2> 715241 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.u.CommitTracker Soft AutoCommit: disabled
>>>    [junit4]   2> 715241 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.u.UpdateLog Could not find max version in index or recent updates,
>>> using new clock 1580641278248353792
>>>    [junit4]   2> 715243 INFO  (searcherExecutor-2416-thread-
>>> 1-processing-n:127.0.0.1:33501_solr x:testCollection1_shard2_replica_n3
>>> s:shard2 c:testCollection1 r:core_node4) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.c.SolrCore [testCollection1_shard2_replica_n3] Registered new
>>> searcher Searcher@15532e2[testCollection1_shard2_replica_n3]
>>> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>>>    [junit4]   2> 715244 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.s.SolrIndexSearcher Opening [Searcher@1202a80[
>>> testCollection1_shard1_replica_n1] main]
>>>    [junit4]   2> 715246 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with
>>> znodeBase: /configs/config
>>>    [junit4]   2> 715246 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
>>> ZooKeeperStorageIO:path=/configs/config
>>>    [junit4]   2> 715248 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
>>>    [junit4]   2> 715249 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
>>>    [junit4]   2> 715249 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and
>>> sync
>>>    [junit4]   2> 715250 INFO  (searcherExecutor-2417-thread-
>>> 1-processing-n:127.0.0.1:33501_solr x:testCollection1_shard1_replica_n1
>>> s:shard1 c:testCollection1 r:core_node2) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.c.SolrCore [testCollection1_shard1_replica_n1] Registered new
>>> searcher Searcher@1202a80[testCollection1_shard1_replica_n1]
>>> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>>>    [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33501/solr/
>>> testCollection1_shard2_replica_n3/
>>>    [junit4]   2> 715250 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.u.UpdateLog Could not find max version in index or recent updates,
>>> using new clock 1580641278257790976
>>>    [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
>>>    [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.c.SyncStrategy https://127.0.0.1:33501/solr/
>>> testCollection1_shard2_replica_n3/ has no replicas
>>>    [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.c.ShardLeaderElectionContext Found all replicas participating in
>>> election, clear LIR
>>>    [junit4]   2> 715253 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.c.ShardLeaderElectionContext I am the new leader:
>>> https://127.0.0.1:33501/solr/testCollection1_shard2_replica_n3/ shard2
>>>    [junit4]   2> 715256 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
>>>    [junit4]   2> 715256 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and
>>> sync
>>>    [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33501/solr/
>>> testCollection1_shard1_replica_n1/
>>>    [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
>>>    [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.c.SyncStrategy https://127.0.0.1:33501/solr/
>>> testCollection1_shard1_replica_n1/ has no replicas
>>>    [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.c.ShardLeaderElectionContext Found all replicas participating in
>>> election, clear LIR
>>>    [junit4]   2> 715260 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.c.ShardLeaderElectionContext I am the new leader:
>>> https://127.0.0.1:33501/solr/testCollection1_shard1_replica_n1/ shard1
>>>    [junit4]   2> 715363 INFO  (zkCallback-1001-thread-2-
>>> processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ]
>>> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
>>> state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json]
>>> for collection [testCollection1] has occurred - updating... (live nodes
>>> size: [1])
>>>    [junit4]   2> 715363 INFO  (zkCallback-1001-thread-1-
>>> processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ]
>>> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
>>> state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json]
>>> for collection [testCollection1] has occurred - updating... (live nodes
>>> size: [1])
>>>    [junit4]   2> 715405 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.c.ZkController I am the leader, no recovery necessary
>>>    [junit4]   2> 715406 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3]
>>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
>>> params={qt=/admin/cores&coreNodeName=core_node4&
>>> collection.configName=config&newCollection=true&name=
>>> testCollection1_shard2_replica_n3&action=CREATE&numShards=2&collection=
>>> testCollection1&shard=shard2&wt=javabin&version=2&replicaType=NRT}
>>> status=0 QTime=1287
>>>    [junit4]   2> 715412 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.c.ZkController I am the leader, no recovery necessary
>>>    [junit4]   2> 715415 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>> c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1]
>>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
>>> params={qt=/admin/cores&coreNodeName=core_node2&
>>> collection.configName=config&newCollection=true&name=
>>> testCollection1_shard1_replica_n1&action=CREATE&numShards=2&collection=
>>> testCollection1&shard=shard1&wt=javabin&version=2&replicaType=NRT}
>>> status=0 QTime=1296
>>>    [junit4]   2> 715419 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for
>>> at most 30 seconds. Check all shard replicas
>>>    [junit4]   2> 715517 INFO  (zkCallback-1001-thread-1-
>>> processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ]
>>> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
>>> state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json]
>>> for collection [testCollection1] has occurred - updating... (live nodes
>>> size: [1])
>>>    [junit4]   2> 715517 INFO  (zkCallback-1001-thread-2-
>>> processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ]
>>> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
>>> state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json]
>>> for collection [testCollection1] has occurred - updating... (live nodes
>>> size: [1])
>>>    [junit4]   2> 715803 INFO  (OverseerCollectionConfigSetPro
>>> cessor-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
>>> [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerTaskQueue Response ZK
>>> path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.
>>> Requestor may have disconnected from ZooKeeper
>>>    [junit4]   2> 716419 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
>>> params={replicationFactor=1&collection.configName=config&
>>> maxShardsPerNode=1000&name=testCollection1&nrtReplicas=1&
>>> action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2622
>>>    [junit4]   2> 716421 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with
>>> params replicationFactor=1&collection.configName=config&
>>> maxShardsPerNode=1000&name=testCollection2&nrtReplicas=1&
>>> action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
>>>    [junit4]   2> 716422 INFO  (OverseerThreadFactory-2413-
>>> thread-2-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr
>>> ] o.a.s.c.CreateCollectionCmd Create collection testCollection2
>>>    [junit4]   2> 716627 INFO  (OverseerStateUpdate-
>>> 98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.c.o.SliceMutator createReplica() {
>>>    [junit4]   2>   "operation":"ADDREPLICA",
>>>    [junit4]   2>   "collection":"testCollection2",
>>>    [junit4]   2>   "shard":"shard1",
>>>    [junit4]   2>   "core":"testCollection2_shard1_replica_n1",
>>>    [junit4]   2>   "state":"down",
>>>    [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
>>>    [junit4]   2>   "type":"NRT"}
>>>    [junit4]   2> 716629 INFO  (OverseerStateUpdate-
>>> 98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.c.o.SliceMutator createReplica() {
>>>    [junit4]   2>   "operation":"ADDREPLICA",
>>>    [junit4]   2>   "collection":"testCollection2",
>>>    [junit4]   2>   "shard":"shard2",
>>>    [junit4]   2>   "core":"testCollection2_shard2_replica_n2",
>>>    [junit4]   2>   "state":"down",
>>>    [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
>>>    [junit4]   2>   "type":"NRT"}
>>>    [junit4]   2> 716831 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.h.a.CoreAdminOperation core create command
>>> qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&
>>> newCollection=true&name=testCollection2_shard2_replica_n2&action=CREATE&
>>> numShards=2&collection=testCollection2&shard=shard2&
>>> wt=javabin&version=2&replicaType=NRT
>>>    [junit4]   2> 716833 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.h.a.CoreAdminOperation core create command
>>> qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&
>>> newCollection=true&name=testCollection2_shard1_replica_n1&action=CREATE&
>>> numShards=2&collection=testCollection2&shard=shard1&
>>> wt=javabin&version=2&replicaType=NRT
>>>    [junit4]   2> 716937 INFO  (zkCallback-1001-thread-1-
>>> processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ]
>>> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
>>> state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json]
>>> for collection [testCollection2] has occurred - updating... (live nodes
>>> size: [1])
>>>    [junit4]   2> 716937 INFO  (zkCallback-1001-thread-2-
>>> processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ]
>>> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
>>> state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json]
>>> for collection [testCollection2] has occurred - updating... (live nodes
>>> size: [1])
>>>    [junit4]   2> 717844 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
>>>    [junit4]   2> 717844 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
>>>    [junit4]   2> 717870 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.s.IndexSchema [testCollection2_shard1_replica_n1] Schema
>>> name=minimal
>>>    [junit4]   2> 717870 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.s.IndexSchema [testCollection2_shard2_replica_n2] Schema
>>> name=minimal
>>>    [junit4]   2> 717874 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
>>>    [junit4]   2> 717874 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
>>>    [junit4]   2> 717874 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.c.CoreContainer Creating SolrCore 'testCollection2_shard1_replica_n1'
>>> using configuration from collection testCollection2, trusted=true
>>>    [junit4]   2> 717874 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.c.CoreContainer Creating SolrCore 'testCollection2_shard2_replica_n2'
>>> using configuration from collection testCollection2, trusted=true
>>>    [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.core.
>>> testCollection2.shard2.replica_n2' (registry 'solr.core.testCollection2.shard2.replica_n2')
>>> enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>>>    [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.core.
>>> testCollection2.shard1.replica_n1' (registry 'solr.core.testCollection2.shard1.replica_n1')
>>> enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>>>    [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
>>>    [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
>>>    [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.c.SolrCore [[testCollection2_shard2_replica_n2] ] Opening new
>>> SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/
>>> build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_
>>> D426A0545BC5456-001/tempDir-001/node1/testCollection2_shard2_replica_n2],
>>> dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-
>>> Linux/solr/build/solr-core/test/J1/temp/solr.cloud.
>>> LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-
>>> 001/node1/./testCollection2_shard2_replica_n2/data/]
>>>    [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.c.SolrCore [[testCollection2_shard1_replica_n1] ] Opening new
>>> SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/
>>> build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_
>>> D426A0545BC5456-001/tempDir-001/node1/testCollection2_shard1_replica_n1],
>>> dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-
>>> Linux/solr/build/solr-core/test/J1/temp/solr.cloud.
>>> LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-
>>> 001/node1/./testCollection2_shard1_replica_n1/data/]
>>>    [junit4]   2> 717939 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.u.UpdateHandler Using UpdateLog implementation:
>>> org.apache.solr.update.UpdateLog
>>>    [junit4]   2> 717940 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null
>>> defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
>>> numVersionBuckets=65536
>>>    [junit4]   2> 717941 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.u.CommitTracker Hard AutoCommit: disabled
>>>    [junit4]   2> 717941 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.u.CommitTracker Soft AutoCommit: disabled
>>>    [junit4]   2> 717942 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.u.UpdateHandler Using UpdateLog implementation:
>>> org.apache.solr.update.UpdateLog
>>>    [junit4]   2> 717942 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.s.SolrIndexSearcher Opening [Searcher@7b3d53[
>>> testCollection2_shard1_replica_n1] main]
>>>    [junit4]   2> 717942 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null
>>> defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
>>> numVersionBuckets=65536
>>>    [junit4]   2> 717943 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with
>>> znodeBase: /configs/config
>>>    [junit4]   2> 717944 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
>>> ZooKeeperStorageIO:path=/configs/config
>>>    [junit4]   2> 717944 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.u.CommitTracker Hard AutoCommit: disabled
>>>    [junit4]   2> 717944 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.u.CommitTracker Soft AutoCommit: disabled
>>>    [junit4]   2> 717945 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
>>>    [junit4]   2> 717946 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.s.SolrIndexSearcher Opening [Searcher@17e3347[
>>> testCollection2_shard2_replica_n2] main]
>>>    [junit4]   2> 717946 INFO  (searcherExecutor-2426-thread-
>>> 1-processing-n:127.0.0.1:33501_solr x:testCollection2_shard1_replica_n1
>>> s:shard1 c:testCollection2 r:core_node3) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.c.SolrCore [testCollection2_shard1_replica_n1] Registered new
>>> searcher Searcher@7b3d53[testCollection2_shard1_replica_n1]
>>> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>>>    [junit4]   2> 717946 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.u.UpdateLog Could not find max version in index or recent updates,
>>> using new clock 1580641281084751872
>>>    [junit4]   2> 717946 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with
>>> znodeBase: /configs/config
>>>    [junit4]   2> 717947 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
>>> ZooKeeperStorageIO:path=/configs/config
>>>    [junit4]   2> 717947 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
>>>    [junit4]   2> 717948 INFO  (searcherExecutor-2427-thread-
>>> 1-processing-n:127.0.0.1:33501_solr x:testCollection2_shard2_replica_n2
>>> s:shard2 c:testCollection2 r:core_node4) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.c.SolrCore [testCollection2_shard2_replica_n2] Registered new
>>> searcher Searcher@17e3347[testCollection2_shard2_replica_n2]
>>> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>>>    [junit4]   2> 717949 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.u.UpdateLog Could not find max version in index or recent updates,
>>> using new clock 1580641281087897600
>>>    [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
>>>    [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and
>>> sync
>>>    [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33501/solr/
>>> testCollection2_shard1_replica_n1/
>>>    [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
>>>    [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.c.SyncStrategy https://127.0.0.1:33501/solr/
>>> testCollection2_shard1_replica_n1/ has no replicas
>>>    [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.c.ShardLeaderElectionContext Found all replicas participating in
>>> election, clear LIR
>>>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
>>>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and
>>> sync
>>>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33501/solr/
>>> testCollection2_shard2_replica_n2/
>>>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
>>>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.c.SyncStrategy https://127.0.0.1:33501/solr/
>>> testCollection2_shard2_replica_n2/ has no replicas
>>>    [junit4]   2> 717976 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.c.ShardLeaderElectionContext Found all replicas participating in
>>> election, clear LIR
>>>    [junit4]   2> 717977 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.c.ShardLeaderElectionContext I am the new leader:
>>> https://127.0.0.1:33501/solr/testCollection2_shard1_replica_n1/ shard1
>>>    [junit4]   2> 717978 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.c.ShardLeaderElectionContext I am the new leader:
>>> https://127.0.0.1:33501/solr/testCollection2_shard2_replica_n2/ shard2
>>>    [junit4]   2> 718081 INFO  (zkCallback-1001-thread-2-
>>> processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ]
>>> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
>>> state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json]
>>> for collection [testCollection2] has occurred - updating... (live nodes
>>> size: [1])
>>>    [junit4]   2> 718082 INFO  (zkCallback-1001-thread-1-
>>> processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ]
>>> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
>>> state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json]
>>> for collection [testCollection2] has occurred - updating... (live nodes
>>> size: [1])
>>>    [junit4]   2> 718129 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.c.ZkController I am the leader, no recovery necessary
>>>    [junit4]   2> 718129 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.c.ZkController I am the leader, no recovery necessary
>>>    [junit4]   2> 718130 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1]
>>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
>>> params={qt=/admin/cores&coreNodeName=core_node3&
>>> collection.configName=config&newCollection=true&name=
>>> testCollection2_shard1_replica_n1&action=CREATE&numShards=2&collection=
>>> testCollection2&shard=shard1&wt=javabin&version=2&replicaType=NRT}
>>> status=0 QTime=1296
>>>    [junit4]   2> 718130 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>> c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2]
>>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
>>> params={qt=/admin/cores&coreNodeName=core_node4&
>>> collection.configName=config&newCollection=true&name=
>>> testCollection2_shard2_replica_n2&action=CREATE&numShards=2&collection=
>>> testCollection2&shard=shard2&wt=javabin&version=2&replicaType=NRT}
>>> status=0 QTime=1299
>>>    [junit4]   2> 718133 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for
>>> at most 30 seconds. Check all shard replicas
>>>    [junit4]   2> 718234 INFO  (zkCallback-1001-thread-1-
>>> processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ]
>>> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
>>> state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json]
>>> for collection [testCollection2] has occurred - updating... (live nodes
>>> size: [1])
>>>    [junit4]   2> 718234 INFO  (zkCallback-1001-thread-3-
>>> processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ]
>>> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
>>> state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json]
>>> for collection [testCollection2] has occurred - updating... (live nodes
>>> size: [1])
>>>    [junit4]   2> 718423 INFO  (OverseerCollectionConfigSetPro
>>> cessor-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
>>> [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerTaskQueue Response ZK
>>> path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.
>>> Requestor may have disconnected from ZooKeeper
>>>    [junit4]   2> 719133 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
>>> params={replicationFactor=1&collection.configName=config&
>>> maxShardsPerNode=1000&name=testCollection2&nrtReplicas=1&
>>> action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2712
>>>    [junit4]   2> 719134 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish
>>> - collection: testCollection1 failOnTimeout:true timeout (sec):30
>>>    [junit4]   2> 719134 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished -
>>> collection: testCollection1
>>>    [junit4]   2> 719134 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish
>>> - collection: testCollection2 failOnTimeout:true timeout (sec):30
>>>    [junit4]   2> 719134 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished -
>>> collection: testCollection2
>>>    [junit4]   2> 719145 INFO  (TEST-LeaderElectionContextKeyTest.test-seed#[D426A0545BC5456])
>>> [    ] o.a.s.SolrTestCaseJ4 ###Starting test
>>>    [junit4]   2> 719153 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.m.SolrMetricManager Closing metric reporters for
>>> registry=solr.core.testCollection1.shard2.replica_n3, tag=null
>>>    [junit4]   2> 719154 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.
>>> reporters.SolrJmxReporter@b5e43b: rootName = solr_33501, domain =
>>> solr.core.testCollection1.shard2.replica_n3, service url = null, agent
>>> id = null] for registry solr.core.testCollection1.shard2.replica_n3 /
>>> com.codahale.metrics.MetricRegistry@100889b
>>>    [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.c.SolrCore [testCollection1_shard2_replica_n3]  CLOSING
>>> SolrCore org.apache.solr.core.SolrCore@8e18e5
>>>    [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.m.SolrMetricManager Closing metric reporters for
>>> registry=solr.core.testCollection1.shard2.replica_n3, tag=9312485
>>>    [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.m.SolrMetricManager Closing metric reporters for
>>> registry=solr.collection.testCollection1.shard2.leader, tag=9312485
>>>    [junit4]   2> 719179 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
>>> params={deleteInstanceDir=false&core=testCollection1_shard2_replica_n3&
>>> deleteDataDir=false&action=UNLOAD&indexInfo=true&wt=
>>> javabin&version=2&deleteIndex=false} status=0 QTime=26
>>>    [junit4]   2> 719181 INFO  (TEST-LeaderElectionContextKeyTest.test-seed#[D426A0545BC5456])
>>> [    ] o.a.s.SolrTestCaseJ4 ###Ending test
>>>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=LeaderElectionContextKeyTest
>>> -Dtests.method=test -Dtests.seed=D426A0545BC5456 -Dtests.multiplier=3
>>> -Dtests.slow=true -Dtests.locale=mk -Dtests.timezone=Canada/Mountain
>>> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>>>    [junit4] FAILURE 0.05s J1 | LeaderElectionContextKeyTest.test <<<
>>>    [junit4]    > Throwable #1: java.lang.AssertionError
>>>    [junit4]    >        at __randomizedtesting.SeedInfo.
>>> seed([D426A0545BC5456:851655DFEB4039AE]:0)
>>>    [junit4]    >        at org.apache.solr.cloud.
>>> LeaderElectionContextKeyTest.test(LeaderElectionContextKeyTest.java:88)
>>>    [junit4]    >        at java.lang.Thread.run(Thread.java:748)
>>>    [junit4]   2> 719191 INFO  (jetty-closer-995-thread-1) [    ]
>>> o.e.j.s.AbstractConnector Stopped ServerConnector@d0d870{SSL,[ssl,
>>> http/1.1]}{127.0.0.1:0}
>>>    [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ]
>>> o.a.s.c.CoreContainer Shutting down CoreContainer instance=31261178
>>>    [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ]
>>> o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node,
>>> tag=null
>>>    [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ]
>>> o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.
>>> reporters.SolrJmxReporter@cfcf8d: rootName = solr_33501, domain =
>>> solr.node, service url = null, agent id = null] for registry solr.node /
>>> com.codahale.metrics.MetricRegistry@8cc4d4
>>>    [junit4]   2> 719200 INFO  (jetty-closer-995-thread-1) [    ]
>>> o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm,
>>> tag=null
>>>    [junit4]   2> 719200 INFO  (jetty-closer-995-thread-1) [    ]
>>> o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.
>>> reporters.SolrJmxReporter@18f30a8: rootName = solr_33501, domain =
>>> solr.jvm, service url = null, agent id = null] for registry solr.jvm /
>>> com.codahale.metrics.MetricRegistry@1bf26ed
>>>    [junit4]   2> 719208 INFO  (jetty-closer-995-thread-1) [    ]
>>> o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty,
>>> tag=null
>>>    [junit4]   2> 719208 INFO  (jetty-closer-995-thread-1) [    ]
>>> o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.
>>> reporters.SolrJmxReporter@f110c8: rootName = solr_33501, domain =
>>> solr.jetty, service url = null, agent id = null] for registry solr.jetty /
>>> com.codahale.metrics.MetricRegistry@192a8b1
>>>    [junit4]   2> 719209 INFO  (jetty-closer-995-thread-1) [    ]
>>> o.a.s.m.SolrMetricManager Closing metric reporters for
>>> registry=solr.cluster, tag=null
>>>    [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-1)
>>> [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2
>>> x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore
>>> [testCollection1_shard1_replica_n1]  CLOSING SolrCore
>>> org.apache.solr.core.SolrCore@16aac00
>>>    [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-1)
>>> [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2
>>> x:testCollection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing
>>> metric reporters for registry=solr.core.testCollection1.shard1.replica_n1,
>>> tag=23768064
>>>    [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-2)
>>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3
>>> x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore
>>> [testCollection2_shard1_replica_n1]  CLOSING SolrCore
>>> org.apache.solr.core.SolrCore@17509de
>>>    [junit4]   2> 719215 INFO  (coreCloseExecutor-2436-thread-3)
>>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4
>>> x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore
>>> [testCollection2_shard2_replica_n2]  CLOSING SolrCore
>>> org.apache.solr.core.SolrCore@140f2a1
>>>    [junit4]   2> 719216 INFO  (coreCloseExecutor-2436-thread-1)
>>> [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2
>>> x:testCollection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing
>>> reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@169f449:
>>> rootName = solr_33501, domain = solr.core.testCollection1.shard1.replica_n1,
>>> service url = null, agent id = null] for registry solr.core.testCollection1.shard1.replica_n1
>>> / com.codahale.metrics.MetricRegistry@121b656
>>>    [junit4]   2> 719230 INFO  (coreCloseExecutor-2436-thread-2)
>>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3
>>> x:testCollection2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing
>>> metric reporters for registry=solr.core.testCollection2.shard1.replica_n1,
>>> tag=24447454
>>>    [junit4]   2> 719230 INFO  (coreCloseExecutor-2436-thread-2)
>>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3
>>> x:testCollection2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing
>>> reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@e8b978:
>>> rootName = solr_33501, domain = solr.core.testCollection2.shard1.replica_n1,
>>> service url = null, agent id = null] for registry solr.core.testCollection2.shard1.replica_n1
>>> / com.codahale.metrics.MetricRegistry@113466a
>>>    [junit4]   2> 719244 INFO  (coreCloseExecutor-2436-thread-3)
>>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4
>>> x:testCollection2_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing
>>> metric reporters for registry=solr.core.testCollection2.shard2.replica_n2,
>>> tag=21033633
>>>    [junit4]   2> 719244 INFO  (coreCloseExecutor-2436-thread-3)
>>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4
>>> x:testCollection2_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing
>>> reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@cfc71:
>>> rootName = solr_33501, domain = solr.core.testCollection2.shard2.replica_n2,
>>> service url = null, agent id = null] for registry solr.core.testCollection2.shard2.replica_n2
>>> / com.codahale.metrics.MetricRegistry@13e256e
>>>    [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-1)
>>> [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2
>>> x:testCollection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing
>>> metric reporters for registry=solr.collection.testCollection1.shard1.leader,
>>> tag=23768064
>>>    [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-2)
>>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3
>>> x:testCollection2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing
>>> metric reporters for registry=solr.collection.testCollection2.shard1.leader,
>>> tag=24447454
>>>    [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-3)
>>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4
>>> x:testCollection2_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing
>>> metric reporters for registry=solr.collection.testCollection2.shard2.leader,
>>> tag=21033633
>>>    [junit4]   2> 719267 INFO  (jetty-closer-995-thread-1) [    ]
>>> o.a.s.c.Overseer Overseer (id=98790079762989059-127.0.0.1:33501_solr-n_0000000000)
>>> closing
>>>    [junit4]   2> 719267 INFO  (OverseerStateUpdate-
>>> 98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr
>>>   ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:33501_solr
>>>    [junit4]   2> 720768 WARN  (zkCallback-1001-thread-3-
>>> processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ]
>>> o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to
>>> ZK: [KeeperErrorCode = Session expired for /live_nodes]
>>>    [junit4]   2> 720768 INFO  (jetty-closer-995-thread-1) [    ]
>>> o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@
>>> 38e81f{/solr,null,UNAVAILABLE}
>>>    [junit4]   2> 720770 ERROR (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so
>>> ZooKeeper server won't take any action on ERROR or SHUTDOWN server state
>>> changes
>>>    [junit4]   2> 720770 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker)
>>> [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:32955 32955
>>>    [junit4]   2> 725792 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer
>>> connecting to 127.0.0.1:32955 32955
>>>    [junit4]   2> 725792 WARN  (Thread-1606) [    ] o.a.s.c.ZkTestServer
>>> Watch limit violations:
>>>    [junit4]   2> Maximum concurrent create/delete watches above limit:
>>>    [junit4]   2>
>>>    [junit4]   2>        2       /solr/aliases.json
>>>    [junit4]   2>
>>>    [junit4]   2> Maximum concurrent data watches above limit:
>>>    [junit4]   2>
>>>    [junit4]   2>        5       /solr/collections/
>>> testCollection2/state.json
>>>    [junit4]   2>        5       /solr/collections/
>>> testCollection1/state.json
>>>    [junit4]   2>        2       /solr/clusterstate.json
>>>    [junit4]   2>        2       /solr/clusterprops.json
>>>    [junit4]   2>
>>>    [junit4]   2> Maximum concurrent children watches above limit:
>>>    [junit4]   2>
>>>    [junit4]   2>        2       /solr/live_nodes
>>>    [junit4]   2>        2       /solr/collections
>>>    [junit4]   2>
>>>    [junit4]   2> NOTE: leaving temporary files on disk at:
>>> /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/
>>> build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_
>>> D426A0545BC5456-001
>>>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {},
>>> docValues:{}, maxPointsInLeafNode=1086, maxMBSortInHeap=5.302125034778907,
>>> sim=RandomSimilarity(queryNorm=true): {}, locale=mk,
>>> timezone=Canada/Mountain
>>>    [junit4]   2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation
>>> 1.8.0_144 (32-bit)/cpus=8,threads=1,free=134427200,total=326377472
>>>    [junit4]   2> NOTE: All tests run in this JVM:
>>> [BasicFunctionalityTest, TestTestInjection, TestConfigSetsAPIZkFailure,
>>> TestNRTOpen, TestPolicyCloud, CollectionsAPIAsyncDistributedZkTest,
>>> DistributedDebugComponentTest, TestFieldTypeCollectionResource,
>>> SearchHandlerTest, TestHighFrequencyDictionaryFactory,
>>> DistributedIntervalFacetingTest, TestReRankQParserPlugin,
>>> DistribJoinFromCollectionTest, TestRTGBase, BigEndianAscendingWordDeserializerTest,
>>> DocValuesMultiTest, SolrCloudExampleTest, TestDistribDocBasedVersion,
>>> CoreAdminHandlerTest, TestCloudSchemaless, TestManagedSynonymFilterFactory,
>>> TestEmbeddedSolrServerAdminHandler, TestDistributedGrouping,
>>> BufferStoreTest, TestMissingGroups, TestPushWriter, PrimUtilsTest,
>>> TestNumericTerms64, ShardRoutingCustomTest, TestSimpleTrackingShardHandler,
>>> AddSchemaFieldsUpdateProcessorFactoryTest,
>>> TestCollectionsAPIViaSolrCloudCluster, TestCloudSearcherWarming,
>>> TestHashPartitioner, ClassificationUpdateProcessorFactoryTest,
>>> TestSolrIndexConfig, RulesTest, JsonLoaderTest, SOLR749Test,
>>> TestHdfsBackupRestoreCore, BadComponentTest, TestBulkSchemaAPI,
>>> BooleanFieldTest, TestSkipOverseerOperations, AsyncCallRequestStatusResponseTest,
>>> TestUseDocValuesAsStored, TestConfigSetImmutable,
>>> TestGeoJSONResponseWriter, TestCollectionAPIs, TestPKIAuthenticationPlugin,
>>> UUIDUpdateProcessorFallbackTest, SystemInfoHandlerTest,
>>> ParsingFieldUpdateProcessorsTest, ExplicitHLLTest,
>>> TestBlendedInfixSuggestions, TestCloudNestedDocsSort, ZkCLITest,
>>> UninvertDocValuesMergePolicyTest, TestRandomRequestDistribution,
>>> LeaderElectionContextKeyTest]
>>>    [junit4] Completed [189/734 (1!)] on J1 in 12.52s, 1 test, 1 failure
>>> <<< FAILURES!
>>>
>>> [...truncated 44564 lines...]
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>> --
>> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>> LinkedIn: http://linkedin.com/in/davidwsmiley | Book: http://www.
>> solrenterprisesearchserver.com
>>
> --
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book: http://www.
> solrenterprisesearchserver.com
>

Re: Geo3dRptTest.testOperations failure

Posted by David Smiley <da...@gmail.com>.
I added toString a few minutes ago (making it to 7.1 release branch).  I
posted reproducible test failure here:
https://issues.apache.org/jira/browse/LUCENE-7990

If you can't get to the problem, so be it.  There's a decent chance this is
just some test error tolerance/precision issue.

Have a nice holiday/vacation.

~ David

On Sun, Oct 8, 2017 at 11:53 AM Ignacio Vera Sequeiros <iv...@eso.org>
wrote:

>
> Hi David,
>
> We should implement toString method indeed. It can just call the toString
> method of the Wrapped shape.
>
> I suspect the error might come from a new shape added to support exact
> circles. If true, I am very interested to understand why is failing.
>
> I am on holidays for the next two weeks and with limited internet
> connection. So if it can wait, I will have a look when I am back.
>
>
> ------------------------------
> *From:* David Smiley <da...@gmail.com>
> *Sent:* Sunday, October 8, 2017 4:19:34 AM
> *To:* dev@lucene.apache.org; Ignacio Vera Sequeiros
> *Subject:* Geo3dRptTest.testOperations failure
>
> I'd file a JIRA issue but it's down at the moment.  Ignacio: any idea
> about this one?
>
> I can see that there is no Geo3dShape.toString; we ought to implement that.
>
> On Sat, Oct 7, 2017 at 7:51 PM Policeman Jenkins Server <
> jenkins@thetaphi.de> wrote:
>
>> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/566/
>> Java: 32bit/jdk1.8.0_144 -client -XX:+UseSerialGC
>>
>> 2 tests failed.
>> FAILED:  org.apache.lucene.spatial.spatial4j.Geo3dRptTest.testOperations
>> {seed=[35F19948C8D0B296:4C8BC51BAD80E140]}
>>
>> Error Message:
>> [Intersects] qIdx:46 Shouldn't match
>> I#1:org.apache.lucene.spatial.spatial4j.Geo3dShape@17ea321b
>> Q:org.apache.lucene.spatial.spatial4j.Geo3dShape@49405867
>>
>> Stack Trace:
>> java.lang.AssertionError: [Intersects] qIdx:46 Shouldn't match
>> I#1:org.apache.lucene.spatial.spatial4j.Geo3dShape@17ea321b
>> Q:org.apache.lucene.spatial.spatial4j.Geo3dShape@49405867
>>         at
>> __randomizedtesting.SeedInfo.seed([35F19948C8D0B296:4C8BC51BAD80E140]:0)
>>         at org.junit.Assert.fail(Assert.java:93)
>>         at
>> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
>>         at
>> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
>>         at
>> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:55)
>>         at
>> org.apache.lucene.spatial.spatial4j.Geo3dRptTest.testOperations(Geo3dRptTest.java:117)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>>         at
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>>         at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>         at
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>>         at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>         at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>         at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>>         at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>>         at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>         at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>         at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>         at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>         at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>         at
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>         at java.lang.Thread.run(Thread.java:748)
>>
>>
>> FAILED:  org.apache.solr.cloud.LeaderElectionContextKeyTest.test
>>
>> Error Message:
>>
>>
>> Stack Trace:
>> java.lang.AssertionError
>>         at
>> __randomizedtesting.SeedInfo.seed([D426A0545BC5456:851655DFEB4039AE]:0)
>>         at org.junit.Assert.fail(Assert.java:92)
>>         at org.junit.Assert.assertTrue(Assert.java:43)
>>         at org.junit.Assert.assertTrue(Assert.java:54)
>>         at
>> org.apache.solr.cloud.LeaderElectionContextKeyTest.test(LeaderElectionContextKeyTest.java:88)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>>         at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>         at
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>>         at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>         at
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>>         at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>         at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>         at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>>         at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>         at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>         at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>         at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>         at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>         at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>         at
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>         at java.lang.Thread.run(Thread.java:748)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 9348 lines...]
>>    [junit4] Suite: org.apache.lucene.spatial.spatial4j.Geo3dRptTest
>>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=Geo3dRptTest
>> -Dtests.method=testOperations -Dtests.seed=35F19948C8D0B296
>> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-MT
>> -Dtests.timezone=Africa/Porto-Novo -Dtests.asserts=true
>> -Dtests.file.encoding=UTF-8
>>    [junit4] FAILURE 3.13s J2 | Geo3dRptTest.testOperations
>> {seed=[35F19948C8D0B296:4C8BC51BAD80E140]} <<<
>>    [junit4]    > Throwable #1: java.lang.AssertionError: [Intersects]
>> qIdx:46 Shouldn't match
>> I#1:org.apache.lucene.spatial.spatial4j.Geo3dShape@17ea321b
>> Q:org.apache.lucene.spatial.spatial4j.Geo3dShape@49405867
>>    [junit4]    >        at
>> __randomizedtesting.SeedInfo.seed([35F19948C8D0B296:4C8BC51BAD80E140]:0)
>>    [junit4]    >        at
>> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
>>    [junit4]    >        at
>> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
>>    [junit4]    >        at
>> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:55)
>>    [junit4]    >        at
>> org.apache.lucene.spatial.spatial4j.Geo3dRptTest.testOperations(Geo3dRptTest.java:117)
>>    [junit4]    >        at java.lang.Thread.run(Thread.java:748)
>>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70):
>> {Geo3dRptTest_rpt=PostingsFormat(name=Memory),
>> id=PostingsFormat(name=Memory)},
>> docValues:{Geo3dRptTest_sdv=DocValuesFormat(name=Lucene70)},
>> maxPointsInLeafNode=478, maxMBSortInHeap=5.983076427221281,
>> sim=RandomSimilarity(queryNorm=false): {}, locale=en-MT,
>> timezone=Africa/Porto-Novo
>>    [junit4]   2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation
>> 1.8.0_144 (32-bit)/cpus=8,threads=1,free=10556296,total=25817088
>>    [junit4]   2> NOTE: All tests run in this JVM: [Geo3dRptTest]
>>    [junit4] Completed [17/23 (1!)] on J2 in 6.97s, 13 tests, 1 failure
>> <<< FAILURES!
>>
>> [...truncated 2808 lines...]
>>    [junit4] Suite: org.apache.solr.cloud.LeaderElectionContextKeyTest
>>    [junit4]   2> 713289 INFO
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.s.SolrTestCaseJ4 SecureRandom sanity checks:
>> test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
>>    [junit4]   2> Creating dataDir:
>> /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/init-core-data-001
>>    [junit4]   2> 713290 WARN
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=9 numCloses=9
>>    [junit4]   2> 713290 INFO
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true)
>> w/NUMERIC_DOCVALUES_SYSPROP=false
>>    [junit4]   2> 713291 INFO
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via:
>> @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN,
>> clientAuth=NaN)
>>    [junit4]   2> 713293 INFO
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in
>> /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001
>>    [junit4]   2> 713294 INFO
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
>>    [junit4]   2> 713294 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer
>> client port:0.0.0.0/0.0.0.0:0
>>    [junit4]   2> 713294 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer
>> Starting server
>>    [junit4]   2> 713296 ERROR (Thread-1606) [    ]
>> o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper
>> server won't take any action on ERROR or SHUTDOWN server state changes
>>    [junit4]   2> 713394 INFO
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.s.c.ZkTestServer start zk server on port:32955
>>    [junit4]   2> 713404 INFO  (jetty-launcher-994-thread-1) [    ]
>> o.e.j.s.Server jetty-9.3.20.v20170531
>>    [junit4]   2> 713405 INFO  (jetty-launcher-994-thread-1) [    ]
>> o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@38e81f
>> {/solr,null,AVAILABLE}
>>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
>> o.e.j.s.AbstractConnector Started ServerConnector@d0d870{SSL,[ssl,
>> http/1.1]}{127.0.0.1:33501}
>>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
>> o.e.j.s.Server Started @714536ms
>>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
>> o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr,
>> hostPort=33501}
>>    [junit4]   2> 713406 ERROR (jetty-launcher-994-thread-1) [    ]
>> o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may
>> be missing or incomplete.
>>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
>> o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr?
>> version 7.1.0
>>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
>> o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port
>> null
>>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
>> o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default
>> config dir: null
>>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
>> o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time:
>> 2017-10-07T22:56:28.682Z
>>    [junit4]   2> 713411 INFO  (jetty-launcher-994-thread-1) [    ]
>> o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>>    [junit4]   2> 713422 INFO  (jetty-launcher-994-thread-1) [    ]
>> o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32955/solr
>>    [junit4]   2> 713481 INFO  (jetty-launcher-994-thread-1)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
>>    [junit4]   2> 713481 INFO  (jetty-launcher-994-thread-1)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerElectionContext I am going
>> to be the leader 127.0.0.1:33501_solr
>>    [junit4]   2> 713482 INFO  (jetty-launcher-994-thread-1)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.Overseer Overseer
>> (id=98790079762989059-127.0.0.1:33501_solr-n_0000000000) starting
>>    [junit4]   2> 713487 INFO  (jetty-launcher-994-thread-1)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.ZkController Register node as live
>> in ZooKeeper:/live_nodes/127.0.0.1:33501_solr
>>    [junit4]   2> 713490 INFO
>> (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes
>> from ZooKeeper... (0) -> (1)
>>    [junit4]   2> 713640 INFO  (jetty-launcher-994-thread-1)
>> [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring
>> for 'solr_33501.solr.node' (registry 'solr.node') enabled at server:
>> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>>    [junit4]   2> 713663 INFO  (jetty-launcher-994-thread-1)
>> [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring
>> for 'solr_33501.solr.jvm' (registry 'solr.jvm') enabled at server:
>> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>>    [junit4]   2> 713663 INFO  (jetty-launcher-994-thread-1)
>> [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring
>> for 'solr_33501.solr.jetty' (registry 'solr.jetty') enabled at server:
>> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>>    [junit4]   2> 713665 INFO  (jetty-launcher-994-thread-1)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core
>> definitions underneath
>> /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/.
>>    [junit4]   2> 713693 INFO
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
>>    [junit4]   2> 713694 INFO
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:32955/solr
>> ready
>>    [junit4]   2> 713797 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with
>> params
>> replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2
>> and sendToOCPQueue=true
>>    [junit4]   2> 713802 INFO
>> (OverseerThreadFactory-2413-thread-1-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.CreateCollectionCmd Create
>> collection testCollection1
>>    [junit4]   2> 713908 INFO
>> (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
>>    [junit4]   2>   "operation":"ADDREPLICA",
>>    [junit4]   2>   "collection":"testCollection1",
>>    [junit4]   2>   "shard":"shard1",
>>    [junit4]   2>   "core":"testCollection1_shard1_replica_n1",
>>    [junit4]   2>   "state":"down",
>>    [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
>>    [junit4]   2>   "type":"NRT"}
>>    [junit4]   2> 713912 INFO
>> (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
>>    [junit4]   2>   "operation":"ADDREPLICA",
>>    [junit4]   2>   "collection":"testCollection1",
>>    [junit4]   2>   "shard":"shard2",
>>    [junit4]   2>   "core":"testCollection1_shard2_replica_n3",
>>    [junit4]   2>   "state":"down",
>>    [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
>>    [junit4]   2>   "type":"NRT"}
>>    [junit4]   2> 714119 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>   ] o.a.s.h.a.CoreAdminOperation core create command
>> qt=/admin/cores&coreNodeName=core_node2&collection.configName=config&newCollection=true&name=testCollection1_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
>>    [junit4]   2> 714119 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>>   ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for
>> 2147483647 <(214)%20748-3647> transient cores
>>    [junit4]   2> 714119 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>>   ] o.a.s.h.a.CoreAdminOperation core create command
>> qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection1_shard2_replica_n3&action=CREATE&numShards=2&collection=testCollection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
>>    [junit4]   2> 714227 INFO
>> (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
>> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/collections/testCollection1/state.json] for collection
>> [testCollection1] has occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 714227 INFO
>> (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
>> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/collections/testCollection1/state.json] for collection
>> [testCollection1] has occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 715133 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene
>> MatchVersion: 7.1.0
>>    [junit4]   2> 715134 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.c.SolrConfig Using Lucene
>> MatchVersion: 7.1.0
>>    [junit4]   2> 715141 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.s.IndexSchema
>> [testCollection1_shard1_replica_n1] Schema name=minimal
>>    [junit4]   2> 715141 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.s.IndexSchema
>> [testCollection1_shard2_replica_n3] Schema name=minimal
>>    [junit4]   2> 715158 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.s.IndexSchema Loaded schema
>> minimal/1.1 with uniqueid field id
>>    [junit4]   2> 715159 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.c.CoreContainer Creating
>> SolrCore 'testCollection1_shard2_replica_n3' using configuration from
>> collection testCollection1, trusted=true
>>    [junit4]   2> 715159 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema
>> minimal/1.1 with uniqueid field id
>>    [junit4]   2> 715159 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.c.CoreContainer Creating
>> SolrCore 'testCollection1_shard1_replica_n1' using configuration from
>> collection testCollection1, trusted=true
>>    [junit4]   2> 715160 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX
>> monitoring for 'solr_33501.solr.core.testCollection1.shard1.replica_n1'
>> (registry 'solr.core.testCollection1.shard1.replica_n1') enabled at server:
>> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>>    [junit4]   2> 715160 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.m.r.SolrJmxReporter JMX
>> monitoring for 'solr_33501.solr.core.testCollection1.shard2.replica_n3'
>> (registry 'solr.core.testCollection1.shard2.replica_n3') enabled at server:
>> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>>    [junit4]   2> 715160 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore
>> solr.RecoveryStrategy.Builder
>>    [junit4]   2> 715161 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore
>> solr.RecoveryStrategy.Builder
>>    [junit4]   2> 715161 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore
>> [[testCollection1_shard1_replica_n1] ] Opening new SolrCore at
>> [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection1_shard1_replica_n1],
>> dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection1_shard1_replica_n1/data/]
>>    [junit4]   2> 715161 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore
>> [[testCollection1_shard2_replica_n3] ] Opening new SolrCore at
>> [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection1_shard2_replica_n3],
>> dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection1_shard2_replica_n3/data/]
>>    [junit4]   2> 715228 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateHandler Using UpdateLog
>> implementation: org.apache.solr.update.UpdateLog
>>    [junit4]   2> 715228 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateLog Initializing
>> UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100
>> maxNumLogsToKeep=10 numVersionBuckets=65536
>>    [junit4]   2> 715230 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.u.CommitTracker Hard AutoCommit:
>> disabled
>>    [junit4]   2> 715230 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.u.CommitTracker Soft AutoCommit:
>> disabled
>>    [junit4]   2> 715233 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.s.SolrIndexSearcher Opening
>> [Searcher@15532e2[testCollection1_shard2_replica_n3] main]
>>    [junit4]   2> 715238 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.r.ManagedResourceStorage
>> Configured ZooKeeperStorageIO with znodeBase: /configs/config
>>    [junit4]   2> 715239 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog
>> implementation: org.apache.solr.update.UpdateLog
>>    [junit4]   2> 715239 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing
>> UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100
>> maxNumLogsToKeep=10 numVersionBuckets=65536
>>    [junit4]   2> 715240 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.r.ManagedResourceStorage Loaded
>> null at path _rest_managed.json using
>> ZooKeeperStorageIO:path=/configs/config
>>    [junit4]   2> 715241 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit:
>> disabled
>>    [junit4]   2> 715241 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.h.ReplicationHandler Commits
>> will be reserved for 10000ms.
>>    [junit4]   2> 715241 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit:
>> disabled
>>    [junit4]   2> 715241 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateLog Could not find max
>> version in index or recent updates, using new clock 1580641278248353792
>>    [junit4]   2> 715243 INFO
>> (searcherExecutor-2416-thread-1-processing-n:127.0.0.1:33501_solr
>> x:testCollection1_shard2_replica_n3 s:shard2 c:testCollection1
>> r:core_node4) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2
>> r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore
>> [testCollection1_shard2_replica_n3] Registered new searcher Searcher@15532e2[testCollection1_shard2_replica_n3]
>> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>>    [junit4]   2> 715244 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening
>> [Searcher@1202a80[testCollection1_shard1_replica_n1] main]
>>    [junit4]   2> 715246 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage
>> Configured ZooKeeperStorageIO with znodeBase: /configs/config
>>    [junit4]   2> 715246 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded
>> null at path _rest_managed.json using
>> ZooKeeperStorageIO:path=/configs/config
>>    [junit4]   2> 715248 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits
>> will be reserved for 10000ms.
>>    [junit4]   2> 715249 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext
>> Enough replicas found to continue.
>>    [junit4]   2> 715249 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext I
>> may be the new leader - try and sync
>>    [junit4]   2> 715250 INFO
>> (searcherExecutor-2417-thread-1-processing-n:127.0.0.1:33501_solr
>> x:testCollection1_shard1_replica_n1 s:shard1 c:testCollection1
>> r:core_node2) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1
>> r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore
>> [testCollection1_shard1_replica_n1] Registered new searcher Searcher@1202a80[testCollection1_shard1_replica_n1]
>> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>>    [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy Sync replicas to
>> https://127.0.0.1:33501/solr/testCollection1_shard2_replica_n3/
>>    [junit4]   2> 715250 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max
>> version in index or recent updates, using new clock 1580641278257790976
>>    [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy Sync Success -
>> now sync replicas to me
>>    [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy
>> https://127.0.0.1:33501/solr/testCollection1_shard2_replica_n3/ has no
>> replicas
>>    [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext
>> Found all replicas participating in election, clear LIR
>>    [junit4]   2> 715253 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext I
>> am the new leader:
>> https://127.0.0.1:33501/solr/testCollection1_shard2_replica_n3/ shard2
>>    [junit4]   2> 715256 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext
>> Enough replicas found to continue.
>>    [junit4]   2> 715256 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I
>> may be the new leader - try and sync
>>    [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to
>> https://127.0.0.1:33501/solr/testCollection1_shard1_replica_n1/
>>    [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success -
>> now sync replicas to me
>>    [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy
>> https://127.0.0.1:33501/solr/testCollection1_shard1_replica_n1/ has no
>> replicas
>>    [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext
>> Found all replicas participating in election, clear LIR
>>    [junit4]   2> 715260 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I
>> am the new leader:
>> https://127.0.0.1:33501/solr/testCollection1_shard1_replica_n1/ shard1
>>    [junit4]   2> 715363 INFO
>> (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
>> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/collections/testCollection1/state.json] for collection
>> [testCollection1] has occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 715363 INFO
>> (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
>> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/collections/testCollection1/state.json] for collection
>> [testCollection1] has occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 715405 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.c.ZkController I am the leader,
>> no recovery necessary
>>    [junit4]   2> 715406 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard2 r:core_node4
>> x:testCollection1_shard2_replica_n3] o.a.s.s.HttpSolrCall [admin]
>> webapp=null path=/admin/cores
>> params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection1_shard2_replica_n3&action=CREATE&numShards=2&collection=testCollection1&shard=shard2&wt=javabin&version=2&replicaType=NRT}
>> status=0 QTime=1287
>>    [junit4]   2> 715412 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.c.ZkController I am the leader,
>> no recovery necessary
>>    [junit4]   2> 715415 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>> c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin]
>> webapp=null path=/admin/cores
>> params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=config&newCollection=true&name=testCollection1_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection1&shard=shard1&wt=javabin&version=2&replicaType=NRT}
>> status=0 QTime=1296
>>    [junit4]   2> 715419 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>   ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for
>> at most 30 seconds. Check all shard replicas
>>    [junit4]   2> 715517 INFO
>> (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
>> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/collections/testCollection1/state.json] for collection
>> [testCollection1] has occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 715517 INFO
>> (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
>> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/collections/testCollection1/state.json] for collection
>> [testCollection1] has occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 715803 INFO
>> (OverseerCollectionConfigSetProcessor-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path:
>> /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor
>> may have disconnected from ZooKeeper
>>    [junit4]   2> 716419 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
>> params={replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2}
>> status=0 QTime=2622
>>    [junit4]   2> 716421 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr
>>   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with
>> params
>> replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection2&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2
>> and sendToOCPQueue=true
>>    [junit4]   2> 716422 INFO
>> (OverseerThreadFactory-2413-thread-2-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.CreateCollectionCmd Create
>> collection testCollection2
>>    [junit4]   2> 716627 INFO
>> (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
>>    [junit4]   2>   "operation":"ADDREPLICA",
>>    [junit4]   2>   "collection":"testCollection2",
>>    [junit4]   2>   "shard":"shard1",
>>    [junit4]   2>   "core":"testCollection2_shard1_replica_n1",
>>    [junit4]   2>   "state":"down",
>>    [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
>>    [junit4]   2>   "type":"NRT"}
>>    [junit4]   2> 716629 INFO
>> (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
>>    [junit4]   2>   "operation":"ADDREPLICA",
>>    [junit4]   2>   "collection":"testCollection2",
>>    [junit4]   2>   "shard":"shard2",
>>    [junit4]   2>   "core":"testCollection2_shard2_replica_n2",
>>    [junit4]   2>   "state":"down",
>>    [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
>>    [junit4]   2>   "type":"NRT"}
>>    [junit4]   2> 716831 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>>   ] o.a.s.h.a.CoreAdminOperation core create command
>> qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection2_shard2_replica_n2&action=CREATE&numShards=2&collection=testCollection2&shard=shard2&wt=javabin&version=2&replicaType=NRT
>>    [junit4]   2> 716833 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>>   ] o.a.s.h.a.CoreAdminOperation core create command
>> qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=testCollection2_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection2&shard=shard1&wt=javabin&version=2&replicaType=NRT
>>    [junit4]   2> 716937 INFO
>> (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
>> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/collections/testCollection2/state.json] for collection
>> [testCollection2] has occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 716937 INFO
>> (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
>> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/collections/testCollection2/state.json] for collection
>> [testCollection2] has occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 717844 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene
>> MatchVersion: 7.1.0
>>    [junit4]   2> 717844 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene
>> MatchVersion: 7.1.0
>>    [junit4]   2> 717870 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.s.IndexSchema
>> [testCollection2_shard1_replica_n1] Schema name=minimal
>>    [junit4]   2> 717870 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.s.IndexSchema
>> [testCollection2_shard2_replica_n2] Schema name=minimal
>>    [junit4]   2> 717874 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema
>> minimal/1.1 with uniqueid field id
>>    [junit4]   2> 717874 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema
>> minimal/1.1 with uniqueid field id
>>    [junit4]   2> 717874 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.c.CoreContainer Creating
>> SolrCore 'testCollection2_shard1_replica_n1' using configuration from
>> collection testCollection2, trusted=true
>>    [junit4]   2> 717874 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.c.CoreContainer Creating
>> SolrCore 'testCollection2_shard2_replica_n2' using configuration from
>> collection testCollection2, trusted=true
>>    [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX
>> monitoring for 'solr_33501.solr.core.testCollection2.shard2.replica_n2'
>> (registry 'solr.core.testCollection2.shard2.replica_n2') enabled at server:
>> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>>    [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX
>> monitoring for 'solr_33501.solr.core.testCollection2.shard1.replica_n1'
>> (registry 'solr.core.testCollection2.shard1.replica_n1') enabled at server:
>> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>>    [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore
>> solr.RecoveryStrategy.Builder
>>    [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore
>> solr.RecoveryStrategy.Builder
>>    [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore
>> [[testCollection2_shard2_replica_n2] ] Opening new SolrCore at
>> [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection2_shard2_replica_n2],
>> dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection2_shard2_replica_n2/data/]
>>    [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore
>> [[testCollection2_shard1_replica_n1] ] Opening new SolrCore at
>> [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection2_shard1_replica_n1],
>> dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection2_shard1_replica_n1/data/]
>>    [junit4]   2> 717939 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog
>> implementation: org.apache.solr.update.UpdateLog
>>    [junit4]   2> 717940 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateLog Initializing
>> UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100
>> maxNumLogsToKeep=10 numVersionBuckets=65536
>>    [junit4]   2> 717941 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit:
>> disabled
>>    [junit4]   2> 717941 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit:
>> disabled
>>    [junit4]   2> 717942 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog
>> implementation: org.apache.solr.update.UpdateLog
>>    [junit4]   2> 717942 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening
>> [Searcher@7b3d53[testCollection2_shard1_replica_n1] main]
>>    [junit4]   2> 717942 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateLog Initializing
>> UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100
>> maxNumLogsToKeep=10 numVersionBuckets=65536
>>    [junit4]   2> 717943 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage
>> Configured ZooKeeperStorageIO with znodeBase: /configs/config
>>    [junit4]   2> 717944 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded
>> null at path _rest_managed.json using
>> ZooKeeperStorageIO:path=/configs/config
>>    [junit4]   2> 717944 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit:
>> disabled
>>    [junit4]   2> 717944 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit:
>> disabled
>>    [junit4]   2> 717945 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits
>> will be reserved for 10000ms.
>>    [junit4]   2> 717946 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening
>> [Searcher@17e3347[testCollection2_shard2_replica_n2] main]
>>    [junit4]   2> 717946 INFO
>> (searcherExecutor-2426-thread-1-processing-n:127.0.0.1:33501_solr
>> x:testCollection2_shard1_replica_n1 s:shard1 c:testCollection2
>> r:core_node3) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1
>> r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore
>> [testCollection2_shard1_replica_n1] Registered new searcher Searcher@7b3d53[testCollection2_shard1_replica_n1]
>> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>>    [junit4]   2> 717946 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max
>> version in index or recent updates, using new clock 1580641281084751872
>>    [junit4]   2> 717946 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.r.ManagedResourceStorage
>> Configured ZooKeeperStorageIO with znodeBase: /configs/config
>>    [junit4]   2> 717947 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded
>> null at path _rest_managed.json using
>> ZooKeeperStorageIO:path=/configs/config
>>    [junit4]   2> 717947 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits
>> will be reserved for 10000ms.
>>    [junit4]   2> 717948 INFO
>> (searcherExecutor-2427-thread-1-processing-n:127.0.0.1:33501_solr
>> x:testCollection2_shard2_replica_n2 s:shard2 c:testCollection2
>> r:core_node4) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2
>> r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore
>> [testCollection2_shard2_replica_n2] Registered new searcher Searcher@17e3347[testCollection2_shard2_replica_n2]
>> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>>    [junit4]   2> 717949 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max
>> version in index or recent updates, using new clock 1580641281087897600
>>    [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext
>> Enough replicas found to continue.
>>    [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I
>> may be the new leader - try and sync
>>    [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to
>> https://127.0.0.1:33501/solr/testCollection2_shard1_replica_n1/
>>    [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success -
>> now sync replicas to me
>>    [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy
>> https://127.0.0.1:33501/solr/testCollection2_shard1_replica_n1/ has no
>> replicas
>>    [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext
>> Found all replicas participating in election, clear LIR
>>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext
>> Enough replicas found to continue.
>>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I
>> may be the new leader - try and sync
>>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to
>> https://127.0.0.1:33501/solr/testCollection2_shard2_replica_n2/
>>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success -
>> now sync replicas to me
>>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy
>> https://127.0.0.1:33501/solr/testCollection2_shard2_replica_n2/ has no
>> replicas
>>    [junit4]   2> 717976 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext
>> Found all replicas participating in election, clear LIR
>>    [junit4]   2> 717977 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I
>> am the new leader:
>> https://127.0.0.1:33501/solr/testCollection2_shard1_replica_n1/ shard1
>>    [junit4]   2> 717978 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I
>> am the new leader:
>> https://127.0.0.1:33501/solr/testCollection2_shard2_replica_n2/ shard2
>>    [junit4]   2> 718081 INFO
>> (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
>> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/collections/testCollection2/state.json] for collection
>> [testCollection2] has occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 718082 INFO
>> (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
>> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/collections/testCollection2/state.json] for collection
>> [testCollection2] has occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 718129 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.c.ZkController I am the leader,
>> no recovery necessary
>>    [junit4]   2> 718129 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.c.ZkController I am the leader,
>> no recovery necessary
>>    [junit4]   2> 718130 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin]
>> webapp=null path=/admin/cores
>> params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=testCollection2_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection2&shard=shard1&wt=javabin&version=2&replicaType=NRT}
>> status=0 QTime=1296
>>    [junit4]   2> 718130 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>> c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin]
>> webapp=null path=/admin/cores
>> params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection2_shard2_replica_n2&action=CREATE&numShards=2&collection=testCollection2&shard=shard2&wt=javabin&version=2&replicaType=NRT}
>> status=0 QTime=1299
>>    [junit4]   2> 718133 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr
>>   ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for
>> at most 30 seconds. Check all shard replicas
>>    [junit4]   2> 718234 INFO
>> (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
>> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/collections/testCollection2/state.json] for collection
>> [testCollection2] has occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 718234 INFO
>> (zkCallback-1001-thread-3-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
>> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/collections/testCollection2/state.json] for collection
>> [testCollection2] has occurred - updating... (live nodes size: [1])
>>    [junit4]   2> 718423 INFO
>> (OverseerCollectionConfigSetProcessor-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path:
>> /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor
>> may have disconnected from ZooKeeper
>>    [junit4]   2> 719133 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr
>>   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
>> params={replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection2&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2}
>> status=0 QTime=2712
>>    [junit4]   2> 719134 INFO
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish -
>> collection: testCollection1 failOnTimeout:true timeout (sec):30
>>    [junit4]   2> 719134 INFO
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection:
>> testCollection1
>>    [junit4]   2> 719134 INFO
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish -
>> collection: testCollection2 failOnTimeout:true timeout (sec):30
>>    [junit4]   2> 719134 INFO
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection:
>> testCollection2
>>    [junit4]   2> 719145 INFO
>> (TEST-LeaderElectionContextKeyTest.test-seed#[D426A0545BC5456]) [    ]
>> o.a.s.SolrTestCaseJ4 ###Starting test
>>    [junit4]   2> 719153 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>   ] o.a.s.m.SolrMetricManager Closing metric reporters for
>> registry=solr.core.testCollection1.shard2.replica_n3, tag=null
>>    [junit4]   2> 719154 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>   ] o.a.s.m.r.SolrJmxReporter Closing reporter
>> [org.apache.solr.metrics.reporters.SolrJmxReporter@b5e43b: rootName =
>> solr_33501, domain = solr.core.testCollection1.shard2.replica_n3, service
>> url = null, agent id = null] for registry
>> solr.core.testCollection1.shard2.replica_n3 /
>> com.codahale.metrics.MetricRegistry@100889b
>>    [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>   ] o.a.s.c.SolrCore [testCollection1_shard2_replica_n3]  CLOSING SolrCore
>> org.apache.solr.core.SolrCore@8e18e5
>>    [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>   ] o.a.s.m.SolrMetricManager Closing metric reporters for
>> registry=solr.core.testCollection1.shard2.replica_n3, tag=9312485
>>    [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>   ] o.a.s.m.SolrMetricManager Closing metric reporters for
>> registry=solr.collection.testCollection1.shard2.leader, tag=9312485
>>    [junit4]   2> 719179 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>>   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
>> params={deleteInstanceDir=false&core=testCollection1_shard2_replica_n3&deleteDataDir=false&action=UNLOAD&indexInfo=true&wt=javabin&version=2&deleteIndex=false}
>> status=0 QTime=26
>>    [junit4]   2> 719181 INFO
>> (TEST-LeaderElectionContextKeyTest.test-seed#[D426A0545BC5456]) [    ]
>> o.a.s.SolrTestCaseJ4 ###Ending test
>>    [junit4]   2> NOTE: reproduce with: ant test
>> -Dtestcase=LeaderElectionContextKeyTest -Dtests.method=test
>> -Dtests.seed=D426A0545BC5456 -Dtests.multiplier=3 -Dtests.slow=true
>> -Dtests.locale=mk -Dtests.timezone=Canada/Mountain -Dtests.asserts=true
>> -Dtests.file.encoding=US-ASCII
>>    [junit4] FAILURE 0.05s J1 | LeaderElectionContextKeyTest.test <<<
>>    [junit4]    > Throwable #1: java.lang.AssertionError
>>    [junit4]    >        at
>> __randomizedtesting.SeedInfo.seed([D426A0545BC5456:851655DFEB4039AE]:0)
>>    [junit4]    >        at
>> org.apache.solr.cloud.LeaderElectionContextKeyTest.test(LeaderElectionContextKeyTest.java:88)
>>    [junit4]    >        at java.lang.Thread.run(Thread.java:748)
>>    [junit4]   2> 719191 INFO  (jetty-closer-995-thread-1) [    ]
>> o.e.j.s.AbstractConnector Stopped ServerConnector@d0d870{SSL,[ssl,
>> http/1.1]}{127.0.0.1:0}
>>    [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ]
>> o.a.s.c.CoreContainer Shutting down CoreContainer instance=31261178
>>    [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ]
>> o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node,
>> tag=null
>>    [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ]
>> o.a.s.m.r.SolrJmxReporter Closing reporter
>> [org.apache.solr.metrics.reporters.SolrJmxReporter@cfcf8d: rootName =
>> solr_33501, domain = solr.node, service url = null, agent id = null] for
>> registry solr.node / com.codahale.metrics.MetricRegistry@8cc4d4
>>    [junit4]   2> 719200 INFO  (jetty-closer-995-thread-1) [    ]
>> o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm,
>> tag=null
>>    [junit4]   2> 719200 INFO  (jetty-closer-995-thread-1) [    ]
>> o.a.s.m.r.SolrJmxReporter Closing reporter
>> [org.apache.solr.metrics.reporters.SolrJmxReporter@18f30a8: rootName =
>> solr_33501, domain = solr.jvm, service url = null, agent id = null] for
>> registry solr.jvm / com.codahale.metrics.MetricRegistry@1bf26ed
>>    [junit4]   2> 719208 INFO  (jetty-closer-995-thread-1) [    ]
>> o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty,
>> tag=null
>>    [junit4]   2> 719208 INFO  (jetty-closer-995-thread-1) [    ]
>> o.a.s.m.r.SolrJmxReporter Closing reporter
>> [org.apache.solr.metrics.reporters.SolrJmxReporter@f110c8: rootName =
>> solr_33501, domain = solr.jetty, service url = null, agent id = null] for
>> registry solr.jetty / com.codahale.metrics.MetricRegistry@192a8b1
>>    [junit4]   2> 719209 INFO  (jetty-closer-995-thread-1) [    ]
>> o.a.s.m.SolrMetricManager Closing metric reporters for
>> registry=solr.cluster, tag=null
>>    [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-1)
>> [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore
>> [testCollection1_shard1_replica_n1]  CLOSING SolrCore
>> org.apache.solr.core.SolrCore@16aac00
>>    [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-1)
>> [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing
>> metric reporters for registry=solr.core.testCollection1.shard1.replica_n1,
>> tag=23768064
>>    [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-2)
>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore
>> [testCollection2_shard1_replica_n1]  CLOSING SolrCore
>> org.apache.solr.core.SolrCore@17509de
>>    [junit4]   2> 719215 INFO  (coreCloseExecutor-2436-thread-3)
>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore
>> [testCollection2_shard2_replica_n2]  CLOSING SolrCore
>> org.apache.solr.core.SolrCore@140f2a1
>>    [junit4]   2> 719216 INFO  (coreCloseExecutor-2436-thread-1)
>> [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing
>> reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@169f449:
>> rootName = solr_33501, domain =
>> solr.core.testCollection1.shard1.replica_n1, service url = null, agent id =
>> null] for registry solr.core.testCollection1.shard1.replica_n1 /
>> com.codahale.metrics.MetricRegistry@121b656
>>    [junit4]   2> 719230 INFO  (coreCloseExecutor-2436-thread-2)
>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing
>> metric reporters for registry=solr.core.testCollection2.shard1.replica_n1,
>> tag=24447454
>>    [junit4]   2> 719230 INFO  (coreCloseExecutor-2436-thread-2)
>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing
>> reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@e8b978:
>> rootName = solr_33501, domain =
>> solr.core.testCollection2.shard1.replica_n1, service url = null, agent id =
>> null] for registry solr.core.testCollection2.shard1.replica_n1 /
>> com.codahale.metrics.MetricRegistry@113466a
>>    [junit4]   2> 719244 INFO  (coreCloseExecutor-2436-thread-3)
>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing
>> metric reporters for registry=solr.core.testCollection2.shard2.replica_n2,
>> tag=21033633
>>    [junit4]   2> 719244 INFO  (coreCloseExecutor-2436-thread-3)
>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing
>> reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@cfc71:
>> rootName = solr_33501, domain =
>> solr.core.testCollection2.shard2.replica_n2, service url = null, agent id =
>> null] for registry solr.core.testCollection2.shard2.replica_n2 /
>> com.codahale.metrics.MetricRegistry@13e256e
>>    [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-1)
>> [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2
>> x:testCollection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing
>> metric reporters for
>> registry=solr.collection.testCollection1.shard1.leader, tag=23768064
>>    [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-2)
>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3
>> x:testCollection2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing
>> metric reporters for
>> registry=solr.collection.testCollection2.shard1.leader, tag=24447454
>>    [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-3)
>> [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4
>> x:testCollection2_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing
>> metric reporters for
>> registry=solr.collection.testCollection2.shard2.leader, tag=21033633
>>    [junit4]   2> 719267 INFO  (jetty-closer-995-thread-1) [    ]
>> o.a.s.c.Overseer Overseer
>> (id=98790079762989059-127.0.0.1:33501_solr-n_0000000000) closing
>>    [junit4]   2> 719267 INFO
>> (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.Overseer Overseer Loop exiting :
>> 127.0.0.1:33501_solr
>>    [junit4]   2> 720768 WARN
>> (zkCallback-1001-thread-3-processing-n:127.0.0.1:33501_solr)
>> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch
>> triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired
>> for /live_nodes]
>>    [junit4]   2> 720768 INFO  (jetty-closer-995-thread-1) [    ]
>> o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@38e81f
>> {/solr,null,UNAVAILABLE}
>>    [junit4]   2> 720770 ERROR
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper
>> server won't take any action on ERROR or SHUTDOWN server state changes
>>    [junit4]   2> 720770 INFO
>> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
>> o.a.s.c.ZkTestServer connecting to 127.0.0.1:32955 32955
>>    [junit4]   2> 725792 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer
>> connecting to 127.0.0.1:32955 32955
>>    [junit4]   2> 725792 WARN  (Thread-1606) [    ] o.a.s.c.ZkTestServer
>> Watch limit violations:
>>    [junit4]   2> Maximum concurrent create/delete watches above limit:
>>    [junit4]   2>
>>    [junit4]   2>        2       /solr/aliases.json
>>    [junit4]   2>
>>    [junit4]   2> Maximum concurrent data watches above limit:
>>    [junit4]   2>
>>    [junit4]   2>        5
>>  /solr/collections/testCollection2/state.json
>>    [junit4]   2>        5
>>  /solr/collections/testCollection1/state.json
>>    [junit4]   2>        2       /solr/clusterstate.json
>>    [junit4]   2>        2       /solr/clusterprops.json
>>    [junit4]   2>
>>    [junit4]   2> Maximum concurrent children watches above limit:
>>    [junit4]   2>
>>    [junit4]   2>        2       /solr/live_nodes
>>    [junit4]   2>        2       /solr/collections
>>    [junit4]   2>
>>    [junit4]   2> NOTE: leaving temporary files on disk at:
>> /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001
>>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {},
>> docValues:{}, maxPointsInLeafNode=1086, maxMBSortInHeap=5.302125034778907,
>> sim=RandomSimilarity(queryNorm=true): {}, locale=mk,
>> timezone=Canada/Mountain
>>    [junit4]   2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation
>> 1.8.0_144 (32-bit)/cpus=8,threads=1,free=134427200,total=326377472
>>    [junit4]   2> NOTE: All tests run in this JVM:
>> [BasicFunctionalityTest, TestTestInjection, TestConfigSetsAPIZkFailure,
>> TestNRTOpen, TestPolicyCloud, CollectionsAPIAsyncDistributedZkTest,
>> DistributedDebugComponentTest, TestFieldTypeCollectionResource,
>> SearchHandlerTest, TestHighFrequencyDictionaryFactory,
>> DistributedIntervalFacetingTest, TestReRankQParserPlugin,
>> DistribJoinFromCollectionTest, TestRTGBase,
>> BigEndianAscendingWordDeserializerTest, DocValuesMultiTest,
>> SolrCloudExampleTest, TestDistribDocBasedVersion, CoreAdminHandlerTest,
>> TestCloudSchemaless, TestManagedSynonymFilterFactory,
>> TestEmbeddedSolrServerAdminHandler, TestDistributedGrouping,
>> BufferStoreTest, TestMissingGroups, TestPushWriter, PrimUtilsTest,
>> TestNumericTerms64, ShardRoutingCustomTest, TestSimpleTrackingShardHandler,
>> AddSchemaFieldsUpdateProcessorFactoryTest,
>> TestCollectionsAPIViaSolrCloudCluster, TestCloudSearcherWarming,
>> TestHashPartitioner, ClassificationUpdateProcessorFactoryTest,
>> TestSolrIndexConfig, RulesTest, JsonLoaderTest, SOLR749Test,
>> TestHdfsBackupRestoreCore, BadComponentTest, TestBulkSchemaAPI,
>> BooleanFieldTest, TestSkipOverseerOperations,
>> AsyncCallRequestStatusResponseTest, TestUseDocValuesAsStored,
>> TestConfigSetImmutable, TestGeoJSONResponseWriter, TestCollectionAPIs,
>> TestPKIAuthenticationPlugin, UUIDUpdateProcessorFallbackTest,
>> SystemInfoHandlerTest, ParsingFieldUpdateProcessorsTest, ExplicitHLLTest,
>> TestBlendedInfixSuggestions, TestCloudNestedDocsSort, ZkCLITest,
>> UninvertDocValuesMergePolicyTest, TestRandomRequestDistribution,
>> LeaderElectionContextKeyTest]
>>    [junit4] Completed [189/734 (1!)] on J1 in 12.52s, 1 test, 1 failure
>> <<< FAILURES!
>>
>> [...truncated 44564 lines...]
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>
> --
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> http://www.solrenterprisesearchserver.com
>
-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

Re: Geo3dRptTest.testOperations failure

Posted by Ignacio Vera Sequeiros <iv...@eso.org>.
Hi David,

We should implement toString method indeed. It can just call the toString method of the Wrapped shape.

I suspect the error might come from a new shape added to support exact circles. If true, I am very interested to understand why is failing.

I am on holidays for the next two weeks and with limited internet connection. So if it can wait, I will have a look when I am back.


________________________________
From: David Smiley <da...@gmail.com>
Sent: Sunday, October 8, 2017 4:19:34 AM
To: dev@lucene.apache.org; Ignacio Vera Sequeiros
Subject: Geo3dRptTest.testOperations failure

I'd file a JIRA issue but it's down at the moment.  Ignacio: any idea about this one?

I can see that there is no Geo3dShape.toString; we ought to implement that.

On Sat, Oct 7, 2017 at 7:51 PM Policeman Jenkins Server <je...@thetaphi.de>> wrote:
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/566/
Java: 32bit/jdk1.8.0_144 -client -XX:+UseSerialGC

2 tests failed.
FAILED:  org.apache.lucene.spatial.spatial4j.Geo3dRptTest.testOperations {seed=[35F19948C8D0B296:4C8BC51BAD80E140]}

Error Message:
[Intersects] qIdx:46 Shouldn't match I#1:org.apache.lucene.spatial.spatial4j.Geo3dShape@17ea321b Q:org.apache.lucene.spatial.spatial4j.Geo3dShape@49405867

Stack Trace:
java.lang.AssertionError: [Intersects] qIdx:46 Shouldn't match I#1:org.apache.lucene.spatial.spatial4j.Geo3dShape@17ea321b Q:org.apache.lucene.spatial.spatial4j.Geo3dShape@49405867
        at __randomizedtesting.SeedInfo.seed([35F19948C8D0B296:4C8BC51BAD80E140]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
        at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
        at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:55)
        at org.apache.lucene.spatial.spatial4j.Geo3dRptTest.testOperations(Geo3dRptTest.java:117)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.LeaderElectionContextKeyTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([D426A0545BC5456:851655DFEB4039AE]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.LeaderElectionContextKeyTest.test(LeaderElectionContextKeyTest.java:88)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 9348 lines...]
   [junit4] Suite: org.apache.lucene.spatial.spatial4j.Geo3dRptTest
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=Geo3dRptTest -Dtests.method=testOperations -Dtests.seed=35F19948C8D0B296 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-MT -Dtests.timezone=Africa/Porto-Novo -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 3.13s J2 | Geo3dRptTest.testOperations {seed=[35F19948C8D0B296:4C8BC51BAD80E140]} <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: [Intersects] qIdx:46 Shouldn't match I#1:org.apache.lucene.spatial.spatial4j.Geo3dShape@17ea321b Q:org.apache.lucene.spatial.spatial4j.Geo3dShape@49405867
   [junit4]    >        at __randomizedtesting.SeedInfo.seed([35F19948C8D0B296:4C8BC51BAD80E140]:0)
   [junit4]    >        at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
   [junit4]    >        at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
   [junit4]    >        at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:55)
   [junit4]    >        at org.apache.lucene.spatial.spatial4j.Geo3dRptTest.testOperations(Geo3dRptTest.java:117)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {Geo3dRptTest_rpt=PostingsFormat(name=Memory), id=PostingsFormat(name=Memory)}, docValues:{Geo3dRptTest_sdv=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=478, maxMBSortInHeap=5.983076427221281, sim=RandomSimilarity(queryNorm=false): {}, locale=en-MT, timezone=Africa/Porto-Novo
   [junit4]   2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=1,free=10556296,total=25817088
   [junit4]   2> NOTE: All tests run in this JVM: [Geo3dRptTest]
   [junit4] Completed [17/23 (1!)] on J2 in 6.97s, 13 tests, 1 failure <<< FAILURES!

[...truncated 2808 lines...]
   [junit4] Suite: org.apache.solr.cloud.LeaderElectionContextKeyTest
   [junit4]   2> 713289 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/init-core-data-001
   [junit4]   2> 713290 WARN  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=9 numCloses=9
   [junit4]   2> 713290 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 713291 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 713293 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001
   [junit4]   2> 713294 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 713294 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0<http://0.0.0.0/0.0.0.0:0>
   [junit4]   2> 713294 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 713296 ERROR (Thread-1606) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 713394 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:32955
   [junit4]   2> 713404 INFO  (jetty-launcher-994-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 713405 INFO  (jetty-launcher-994-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@38e81f{/solr,null,AVAILABLE}
   [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@d0d870{SSL,[ssl, http/1.1]}{127.0.0.1:33501<http://127.0.0.1:33501>}
   [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ] o.e.j.s.Server Started @714536ms
   [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33501}
   [junit4]   2> 713406 ERROR (jetty-launcher-994-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-07T22:56:28.682Z
   [junit4]   2> 713411 INFO  (jetty-launcher-994-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 713422 INFO  (jetty-launcher-994-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32955/solr<http://127.0.0.1:32955/solr>
   [junit4]   2> 713481 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 713481 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33501_solr
   [junit4]   2> 713482 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.c.Overseer Overseer (id=98790079762989059-127.0.0.1:33501_solr-n_0000000000) starting
   [junit4]   2> 713487 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.<http://127.0.0.>1:33501_solr
   [junit4]   2> 713490 INFO  (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 713640 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
   [junit4]   2> 713663 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
   [junit4]   2> 713663 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
   [junit4]   2> 713665 INFO  (jetty-launcher-994-thread-1) [n:127.0.0.1:33501_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/.
   [junit4]   2> 713693 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 713694 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:32955/solr<http://127.0.0.1:32955/solr> ready
   [junit4]   2> 713797 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 713802 INFO  (OverseerThreadFactory-2413-thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.CreateCollectionCmd Create collection testCollection1
   [junit4]   2> 713908 INFO  (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testCollection1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testCollection1_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 713912 INFO  (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testCollection1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testCollection1_shard2_replica_n3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 714119 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=config&newCollection=true&name=testCollection1_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 714119 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647<tel:(214)%20748-3647> transient cores
   [junit4]   2> 714119 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection1_shard2_replica_n3&action=CREATE&numShards=2&collection=testCollection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 714227 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 714227 INFO  (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 715133 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 715134 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 715141 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.s.IndexSchema [testCollection1_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 715141 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.s.IndexSchema [testCollection1_shard2_replica_n3] Schema name=minimal
   [junit4]   2> 715158 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 715159 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.CoreContainer Creating SolrCore 'testCollection1_shard2_replica_n3' using configuration from collection testCollection1, trusted=true
   [junit4]   2> 715159 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 715159 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testCollection1_shard1_replica_n1' using configuration from collection testCollection1, trusted=true
   [junit4]   2> 715160 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.core.testCollection1.shard1.replica_n1' (registry 'solr.core.testCollection1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
   [junit4]   2> 715160 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.core.testCollection1.shard2.replica_n3' (registry 'solr.core.testCollection1.shard2.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
   [junit4]   2> 715160 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 715161 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 715161 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore [[testCollection1_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection1_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection1_shard1_replica_n1/data/]
   [junit4]   2> 715161 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore [[testCollection1_shard2_replica_n3] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection1_shard2_replica_n3], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection1_shard2_replica_n3/data/]
   [junit4]   2> 715228 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 715228 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 715230 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 715230 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 715233 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.s.SolrIndexSearcher Opening [Searcher@15532e2[testCollection1_shard2_replica_n3] main]
   [junit4]   2> 715238 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 715239 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 715239 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 715240 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 715241 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 715241 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 715241 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 715241 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1580641278248353792
   [junit4]   2> 715243 INFO  (searcherExecutor-2416-thread-1-processing-n:127.0.0.1:33501<http://127.0.0.1:33501>_solr x:testCollection1_shard2_replica_n3 s:shard2 c:testCollection1 r:core_node4) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore [testCollection1_shard2_replica_n3] Registered new searcher Searcher@15532e2[testCollection1_shard2_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 715244 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1202a80[testCollection1_shard1_replica_n1] main]
   [junit4]   2> 715246 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 715246 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 715248 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 715249 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 715249 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 715250 INFO  (searcherExecutor-2417-thread-1-processing-n:127.0.0.1:33501<http://127.0.0.1:33501>_solr x:testCollection1_shard1_replica_n1 s:shard1 c:testCollection1 r:core_node2) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore [testCollection1_shard1_replica_n1] Registered new searcher Searcher@1202a80[testCollection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33501/solr/testCollection1_shard2_replica_n3/
   [junit4]   2> 715250 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1580641278257790976
   [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy https://127.0.0.1:33501/solr/testCollection1_shard2_replica_n3/ has no replicas
   [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 715253 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:33501/solr/testCollection1_shard2_replica_n3/ shard2
   [junit4]   2> 715256 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 715256 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33501/solr/testCollection1_shard1_replica_n1/
   [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:33501/solr/testCollection1_shard1_replica_n1/ has no replicas
   [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 715260 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:33501/solr/testCollection1_shard1_replica_n1/ shard1
   [junit4]   2> 715363 INFO  (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 715363 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 715405 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 715406 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection1_shard2_replica_n3&action=CREATE&numShards=2&collection=testCollection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1287
   [junit4]   2> 715412 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 715415 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=config&newCollection=true&name=testCollection1_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1296
   [junit4]   2> 715419 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 715517 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 715517 INFO  (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 715803 INFO  (OverseerCollectionConfigSetProcessor-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 716419 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2622
   [junit4]   2> 716421 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection2&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 716422 INFO  (OverseerThreadFactory-2413-thread-2-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.CreateCollectionCmd Create collection testCollection2
   [junit4]   2> 716627 INFO  (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testCollection2",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testCollection2_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 716629 INFO  (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testCollection2",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testCollection2_shard2_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 716831 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection2_shard2_replica_n2&action=CREATE&numShards=2&collection=testCollection2&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 716833 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=testCollection2_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection2&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 716937 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 716937 INFO  (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 717844 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 717844 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 717870 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.s.IndexSchema [testCollection2_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 717870 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.s.IndexSchema [testCollection2_shard2_replica_n2] Schema name=minimal
   [junit4]   2> 717874 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 717874 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 717874 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testCollection2_shard1_replica_n1' using configuration from collection testCollection2, trusted=true
   [junit4]   2> 717874 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testCollection2_shard2_replica_n2' using configuration from collection testCollection2, trusted=true
   [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.core.testCollection2.shard2.replica_n2' (registry 'solr.core.testCollection2.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
   [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33501.solr.core.testCollection2.shard1.replica_n1' (registry 'solr.core.testCollection2.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
   [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore [[testCollection2_shard2_replica_n2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection2_shard2_replica_n2], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection2_shard2_replica_n2/data/]
   [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore [[testCollection2_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection2_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection2_shard1_replica_n1/data/]
   [junit4]   2> 717939 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 717940 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 717941 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 717941 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 717942 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 717942 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7b3d53[testCollection2_shard1_replica_n1] main]
   [junit4]   2> 717942 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 717943 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 717944 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 717944 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 717944 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 717945 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 717946 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@17e3347[testCollection2_shard2_replica_n2] main]
   [junit4]   2> 717946 INFO  (searcherExecutor-2426-thread-1-processing-n:127.0.0.1:33501<http://127.0.0.1:33501>_solr x:testCollection2_shard1_replica_n1 s:shard1 c:testCollection2 r:core_node3) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore [testCollection2_shard1_replica_n1] Registered new searcher Searcher@7b3d53[testCollection2_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 717946 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1580641281084751872
   [junit4]   2> 717946 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 717947 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 717947 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 717948 INFO  (searcherExecutor-2427-thread-1-processing-n:127.0.0.1:33501<http://127.0.0.1:33501>_solr x:testCollection2_shard2_replica_n2 s:shard2 c:testCollection2 r:core_node4) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore [testCollection2_shard2_replica_n2] Registered new searcher Searcher@17e3347[testCollection2_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 717949 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1580641281087897600
   [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33501/solr/testCollection2_shard1_replica_n1/
   [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:33501/solr/testCollection2_shard1_replica_n1/ has no replicas
   [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33501/solr/testCollection2_shard2_replica_n2/
   [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy https://127.0.0.1:33501/solr/testCollection2_shard2_replica_n2/ has no replicas
   [junit4]   2> 717976 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 717977 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:33501/solr/testCollection2_shard1_replica_n1/ shard1
   [junit4]   2> 717978 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:33501/solr/testCollection2_shard2_replica_n2/ shard2
   [junit4]   2> 718081 INFO  (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 718082 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 718129 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 718129 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 718130 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=testCollection2_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection2&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1296
   [junit4]   2> 718130 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection2_shard2_replica_n2&action=CREATE&numShards=2&collection=testCollection2&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1299
   [junit4]   2> 718133 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 718234 INFO  (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 718234 INFO  (zkCallback-1001-thread-3-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 718423 INFO  (OverseerCollectionConfigSetProcessor-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 719133 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection2&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2712
   [junit4]   2> 719134 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: testCollection1 failOnTimeout:true timeout (sec):30
   [junit4]   2> 719134 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: testCollection1
   [junit4]   2> 719134 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: testCollection2 failOnTimeout:true timeout (sec):30
   [junit4]   2> 719134 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: testCollection2
   [junit4]   2> 719145 INFO  (TEST-LeaderElectionContextKeyTest.test-seed#[D426A0545BC5456]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 719153 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection1.shard2.replica_n3, tag=null
   [junit4]   2> 719154 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@b5e43b: rootName = solr_33501, domain = solr.core.testCollection1.shard2.replica_n3, service url = null, agent id = null] for registry solr.core.testCollection1.shard2.replica_n3 / com.codahale.metrics.MetricRegistry@100889b
   [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.c.SolrCore [testCollection1_shard2_replica_n3]  CLOSING SolrCore org.apache.solr.core.SolrCore@8e18e5
   [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection1.shard2.replica_n3, tag=9312485
   [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCollection1.shard2.leader, tag=9312485
   [junit4]   2> 719179 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=false&core=testCollection1_shard2_replica_n3&deleteDataDir=false&action=UNLOAD&indexInfo=true&wt=javabin&version=2&deleteIndex=false} status=0 QTime=26
   [junit4]   2> 719181 INFO  (TEST-LeaderElectionContextKeyTest.test-seed#[D426A0545BC5456]) [    ] o.a.s.SolrTestCaseJ4 ###Ending test
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=LeaderElectionContextKeyTest -Dtests.method=test -Dtests.seed=D426A0545BC5456 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mk -Dtests.timezone=Canada/Mountain -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.05s J1 | LeaderElectionContextKeyTest.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    >        at __randomizedtesting.SeedInfo.seed([D426A0545BC5456:851655DFEB4039AE]:0)
   [junit4]    >        at org.apache.solr.cloud.LeaderElectionContextKeyTest.test(LeaderElectionContextKeyTest.java:88)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 719191 INFO  (jetty-closer-995-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@d0d870{SSL,[ssl, http/1.1]}{127.0.0.1:0<http://127.0.0.1:0>}
   [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=31261178
   [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@cfcf8d: rootName = solr_33501, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@8cc4d4
   [junit4]   2> 719200 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 719200 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@18f30a8: rootName = solr_33501, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@1bf26ed
   [junit4]   2> 719208 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 719208 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@f110c8: rootName = solr_33501, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@192a8b1
   [junit4]   2> 719209 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-1) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore [testCollection1_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@16aac00
   [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-1) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection1.shard1.replica_n1, tag=23768064
   [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-2) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore [testCollection2_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@17509de
   [junit4]   2> 719215 INFO  (coreCloseExecutor-2436-thread-3) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore [testCollection2_shard2_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@140f2a1
   [junit4]   2> 719216 INFO  (coreCloseExecutor-2436-thread-1) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@169f449: rootName = solr_33501, domain = solr.core.testCollection1.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.testCollection1.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@121b656
   [junit4]   2> 719230 INFO  (coreCloseExecutor-2436-thread-2) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection2.shard1.replica_n1, tag=24447454
   [junit4]   2> 719230 INFO  (coreCloseExecutor-2436-thread-2) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@e8b978: rootName = solr_33501, domain = solr.core.testCollection2.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.testCollection2.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@113466a
   [junit4]   2> 719244 INFO  (coreCloseExecutor-2436-thread-3) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection2.shard2.replica_n2, tag=21033633
   [junit4]   2> 719244 INFO  (coreCloseExecutor-2436-thread-3) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@cfc71: rootName = solr_33501, domain = solr.core.testCollection2.shard2.replica_n2, service url = null, agent id = null] for registry solr.core.testCollection2.shard2.replica_n2 / com.codahale.metrics.MetricRegistry@13e256e
   [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-1) [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCollection1.shard1.leader, tag=23768064
   [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-2) [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCollection2.shard1.leader, tag=24447454
   [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-3) [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCollection2.shard2.leader, tag=21033633
   [junit4]   2> 719267 INFO  (jetty-closer-995-thread-1) [    ] o.a.s.c.Overseer Overseer (id=98790079762989059-127.0.0.1:33501_solr-n_0000000000) closing
   [junit4]   2> 719267 INFO  (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000) [n:127.0.0.1:33501_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:33501_solr
   [junit4]   2> 720768 WARN  (zkCallback-1001-thread-3-processing-n:127.0.0.1:33501_solr) [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 720768 INFO  (jetty-closer-995-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@38e81f{/solr,null,UNAVAILABLE}
   [junit4]   2> 720770 ERROR (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 720770 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:32955<http://127.0.0.1:32955> 32955
   [junit4]   2> 725792 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:32955<http://127.0.0.1:32955> 32955
   [junit4]   2> 725792 WARN  (Thread-1606) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2>        2       /solr/aliases.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2>        5       /solr/collections/testCollection2/state.json
   [junit4]   2>        5       /solr/collections/testCollection1/state.json
   [junit4]   2>        2       /solr/clusterstate.json
   [junit4]   2>        2       /solr/clusterprops.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2>        2       /solr/live_nodes
   [junit4]   2>        2       /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1086, maxMBSortInHeap=5.302125034778907, sim=RandomSimilarity(queryNorm=true): {}, locale=mk, timezone=Canada/Mountain
   [junit4]   2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=1,free=134427200,total=326377472
   [junit4]   2> NOTE: All tests run in this JVM: [BasicFunctionalityTest, TestTestInjection, TestConfigSetsAPIZkFailure, TestNRTOpen, TestPolicyCloud, CollectionsAPIAsyncDistributedZkTest, DistributedDebugComponentTest, TestFieldTypeCollectionResource, SearchHandlerTest, TestHighFrequencyDictionaryFactory, DistributedIntervalFacetingTest, TestReRankQParserPlugin, DistribJoinFromCollectionTest, TestRTGBase, BigEndianAscendingWordDeserializerTest, DocValuesMultiTest, SolrCloudExampleTest, TestDistribDocBasedVersion, CoreAdminHandlerTest, TestCloudSchemaless, TestManagedSynonymFilterFactory, TestEmbeddedSolrServerAdminHandler, TestDistributedGrouping, BufferStoreTest, TestMissingGroups, TestPushWriter, PrimUtilsTest, TestNumericTerms64, ShardRoutingCustomTest, TestSimpleTrackingShardHandler, AddSchemaFieldsUpdateProcessorFactoryTest, TestCollectionsAPIViaSolrCloudCluster, TestCloudSearcherWarming, TestHashPartitioner, ClassificationUpdateProcessorFactoryTest, TestSolrIndexConfig, RulesTest, JsonLoaderTest, SOLR749Test, TestHdfsBackupRestoreCore, BadComponentTest, TestBulkSchemaAPI, BooleanFieldTest, TestSkipOverseerOperations, AsyncCallRequestStatusResponseTest, TestUseDocValuesAsStored, TestConfigSetImmutable, TestGeoJSONResponseWriter, TestCollectionAPIs, TestPKIAuthenticationPlugin, UUIDUpdateProcessorFallbackTest, SystemInfoHandlerTest, ParsingFieldUpdateProcessorsTest, ExplicitHLLTest, TestBlendedInfixSuggestions, TestCloudNestedDocsSort, ZkCLITest, UninvertDocValuesMergePolicyTest, TestRandomRequestDistribution, LeaderElectionContextKeyTest]
   [junit4] Completed [189/734 (1!)] on J1 in 12.52s, 1 test, 1 failure <<< FAILURES!

[...truncated 44564 lines...]

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org<ma...@lucene.apache.org>
For additional commands, e-mail: dev-help@lucene.apache.org<ma...@lucene.apache.org>
--
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book: http://www.solrenterprisesearchserver.com

Geo3dRptTest.testOperations failure

Posted by David Smiley <da...@gmail.com>.
I'd file a JIRA issue but it's down at the moment.  Ignacio: any idea about
this one?

I can see that there is no Geo3dShape.toString; we ought to implement that.

On Sat, Oct 7, 2017 at 7:51 PM Policeman Jenkins Server <je...@thetaphi.de>
wrote:

> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/566/
> Java: 32bit/jdk1.8.0_144 -client -XX:+UseSerialGC
>
> 2 tests failed.
> FAILED:  org.apache.lucene.spatial.spatial4j.Geo3dRptTest.testOperations
> {seed=[35F19948C8D0B296:4C8BC51BAD80E140]}
>
> Error Message:
> [Intersects] qIdx:46 Shouldn't match
> I#1:org.apache.lucene.spatial.spatial4j.Geo3dShape@17ea321b
> Q:org.apache.lucene.spatial.spatial4j.Geo3dShape@49405867
>
> Stack Trace:
> java.lang.AssertionError: [Intersects] qIdx:46 Shouldn't match
> I#1:org.apache.lucene.spatial.spatial4j.Geo3dShape@17ea321b
> Q:org.apache.lucene.spatial.spatial4j.Geo3dShape@49405867
>         at
> __randomizedtesting.SeedInfo.seed([35F19948C8D0B296:4C8BC51BAD80E140]:0)
>         at org.junit.Assert.fail(Assert.java:93)
>         at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
>         at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
>         at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:55)
>         at
> org.apache.lucene.spatial.spatial4j.Geo3dRptTest.testOperations(Geo3dRptTest.java:117)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>         at java.lang.Thread.run(Thread.java:748)
>
>
> FAILED:  org.apache.solr.cloud.LeaderElectionContextKeyTest.test
>
> Error Message:
>
>
> Stack Trace:
> java.lang.AssertionError
>         at
> __randomizedtesting.SeedInfo.seed([D426A0545BC5456:851655DFEB4039AE]:0)
>         at org.junit.Assert.fail(Assert.java:92)
>         at org.junit.Assert.assertTrue(Assert.java:43)
>         at org.junit.Assert.assertTrue(Assert.java:54)
>         at
> org.apache.solr.cloud.LeaderElectionContextKeyTest.test(LeaderElectionContextKeyTest.java:88)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>         at java.lang.Thread.run(Thread.java:748)
>
>
>
>
> Build Log:
> [...truncated 9348 lines...]
>    [junit4] Suite: org.apache.lucene.spatial.spatial4j.Geo3dRptTest
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=Geo3dRptTest
> -Dtests.method=testOperations -Dtests.seed=35F19948C8D0B296
> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-MT
> -Dtests.timezone=Africa/Porto-Novo -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
>    [junit4] FAILURE 3.13s J2 | Geo3dRptTest.testOperations
> {seed=[35F19948C8D0B296:4C8BC51BAD80E140]} <<<
>    [junit4]    > Throwable #1: java.lang.AssertionError: [Intersects]
> qIdx:46 Shouldn't match
> I#1:org.apache.lucene.spatial.spatial4j.Geo3dShape@17ea321b
> Q:org.apache.lucene.spatial.spatial4j.Geo3dShape@49405867
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([35F19948C8D0B296:4C8BC51BAD80E140]:0)
>    [junit4]    >        at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
>    [junit4]    >        at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
>    [junit4]    >        at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:55)
>    [junit4]    >        at
> org.apache.lucene.spatial.spatial4j.Geo3dRptTest.testOperations(Geo3dRptTest.java:117)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:748)
>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70):
> {Geo3dRptTest_rpt=PostingsFormat(name=Memory),
> id=PostingsFormat(name=Memory)},
> docValues:{Geo3dRptTest_sdv=DocValuesFormat(name=Lucene70)},
> maxPointsInLeafNode=478, maxMBSortInHeap=5.983076427221281,
> sim=RandomSimilarity(queryNorm=false): {}, locale=en-MT,
> timezone=Africa/Porto-Novo
>    [junit4]   2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation
> 1.8.0_144 (32-bit)/cpus=8,threads=1,free=10556296,total=25817088
>    [junit4]   2> NOTE: All tests run in this JVM: [Geo3dRptTest]
>    [junit4] Completed [17/23 (1!)] on J2 in 6.97s, 13 tests, 1 failure <<<
> FAILURES!
>
> [...truncated 2808 lines...]
>    [junit4] Suite: org.apache.solr.cloud.LeaderElectionContextKeyTest
>    [junit4]   2> 713289 INFO
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.s.SolrTestCaseJ4 SecureRandom sanity checks:
> test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
>    [junit4]   2> Creating dataDir:
> /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/init-core-data-001
>    [junit4]   2> 713290 WARN
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=9 numCloses=9
>    [junit4]   2> 713290 INFO
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true)
> w/NUMERIC_DOCVALUES_SYSPROP=false
>    [junit4]   2> 713291 INFO
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via:
> @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN,
> clientAuth=NaN)
>    [junit4]   2> 713293 INFO
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in
> /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001
>    [junit4]   2> 713294 INFO
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
>    [junit4]   2> 713294 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer
> client port:0.0.0.0/0.0.0.0:0
>    [junit4]   2> 713294 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer
> Starting server
>    [junit4]   2> 713296 ERROR (Thread-1606) [    ] o.a.z.s.ZooKeeperServer
> ZKShutdownHandler is not registered, so ZooKeeper server won't take any
> action on ERROR or SHUTDOWN server state changes
>    [junit4]   2> 713394 INFO
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.s.c.ZkTestServer start zk server on port:32955
>    [junit4]   2> 713404 INFO  (jetty-launcher-994-thread-1) [    ]
> o.e.j.s.Server jetty-9.3.20.v20170531
>    [junit4]   2> 713405 INFO  (jetty-launcher-994-thread-1) [    ]
> o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@38e81f
> {/solr,null,AVAILABLE}
>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
> o.e.j.s.AbstractConnector Started ServerConnector@d0d870{SSL,[ssl,
> http/1.1]}{127.0.0.1:33501}
>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
> o.e.j.s.Server Started @714536ms
>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
> o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr,
> hostPort=33501}
>    [junit4]   2> 713406 ERROR (jetty-launcher-994-thread-1) [    ]
> o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may
> be missing or incomplete.
>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
> o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr?
> version 7.1.0
>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
> o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port
> null
>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
> o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default
> config dir: null
>    [junit4]   2> 713406 INFO  (jetty-launcher-994-thread-1) [    ]
> o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time:
> 2017-10-07T22:56:28.682Z
>    [junit4]   2> 713411 INFO  (jetty-launcher-994-thread-1) [    ]
> o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>    [junit4]   2> 713422 INFO  (jetty-launcher-994-thread-1) [    ]
> o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32955/solr
>    [junit4]   2> 713481 INFO  (jetty-launcher-994-thread-1)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
>    [junit4]   2> 713481 INFO  (jetty-launcher-994-thread-1)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerElectionContext I am going
> to be the leader 127.0.0.1:33501_solr
>    [junit4]   2> 713482 INFO  (jetty-launcher-994-thread-1)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.Overseer Overseer
> (id=98790079762989059-127.0.0.1:33501_solr-n_0000000000) starting
>    [junit4]   2> 713487 INFO  (jetty-launcher-994-thread-1)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.ZkController Register node as live
> in ZooKeeper:/live_nodes/127.0.0.1:33501_solr
>    [junit4]   2> 713490 INFO
> (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes
> from ZooKeeper... (0) -> (1)
>    [junit4]   2> 713640 INFO  (jetty-launcher-994-thread-1)
> [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_33501.solr.node' (registry 'solr.node') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>    [junit4]   2> 713663 INFO  (jetty-launcher-994-thread-1)
> [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_33501.solr.jvm' (registry 'solr.jvm') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>    [junit4]   2> 713663 INFO  (jetty-launcher-994-thread-1)
> [n:127.0.0.1:33501_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_33501.solr.jetty' (registry 'solr.jetty') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>    [junit4]   2> 713665 INFO  (jetty-launcher-994-thread-1)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core
> definitions underneath
> /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/.
>    [junit4]   2> 713693 INFO
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
>    [junit4]   2> 713694 INFO
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:32955/solr
> ready
>    [junit4]   2> 713797 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with
> params
> replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2
> and sendToOCPQueue=true
>    [junit4]   2> 713802 INFO
> (OverseerThreadFactory-2413-thread-1-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.CreateCollectionCmd Create
> collection testCollection1
>    [junit4]   2> 713908 INFO
> (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
>    [junit4]   2>   "operation":"ADDREPLICA",
>    [junit4]   2>   "collection":"testCollection1",
>    [junit4]   2>   "shard":"shard1",
>    [junit4]   2>   "core":"testCollection1_shard1_replica_n1",
>    [junit4]   2>   "state":"down",
>    [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
>    [junit4]   2>   "type":"NRT"}
>    [junit4]   2> 713912 INFO
> (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
>    [junit4]   2>   "operation":"ADDREPLICA",
>    [junit4]   2>   "collection":"testCollection1",
>    [junit4]   2>   "shard":"shard2",
>    [junit4]   2>   "core":"testCollection1_shard2_replica_n3",
>    [junit4]   2>   "state":"down",
>    [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
>    [junit4]   2>   "type":"NRT"}
>    [junit4]   2> 714119 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>   ] o.a.s.h.a.CoreAdminOperation core create command
> qt=/admin/cores&coreNodeName=core_node2&collection.configName=config&newCollection=true&name=testCollection1_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
>    [junit4]   2> 714119 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
>   ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for
> 2147483647 <(214)%20748-3647> transient cores
>    [junit4]   2> 714119 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
>   ] o.a.s.h.a.CoreAdminOperation core create command
> qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection1_shard2_replica_n3&action=CREATE&numShards=2&collection=testCollection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
>    [junit4]   2> 714227 INFO
> (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/testCollection1/state.json] for collection
> [testCollection1] has occurred - updating... (live nodes size: [1])
>    [junit4]   2> 714227 INFO
> (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/testCollection1/state.json] for collection
> [testCollection1] has occurred - updating... (live nodes size: [1])
>    [junit4]   2> 715133 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene
> MatchVersion: 7.1.0
>    [junit4]   2> 715134 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.c.SolrConfig Using Lucene
> MatchVersion: 7.1.0
>    [junit4]   2> 715141 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.s.IndexSchema
> [testCollection1_shard1_replica_n1] Schema name=minimal
>    [junit4]   2> 715141 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.s.IndexSchema
> [testCollection1_shard2_replica_n3] Schema name=minimal
>    [junit4]   2> 715158 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.s.IndexSchema Loaded schema
> minimal/1.1 with uniqueid field id
>    [junit4]   2> 715159 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.c.CoreContainer Creating
> SolrCore 'testCollection1_shard2_replica_n3' using configuration from
> collection testCollection1, trusted=true
>    [junit4]   2> 715159 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema
> minimal/1.1 with uniqueid field id
>    [junit4]   2> 715159 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.c.CoreContainer Creating
> SolrCore 'testCollection1_shard1_replica_n1' using configuration from
> collection testCollection1, trusted=true
>    [junit4]   2> 715160 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX
> monitoring for 'solr_33501.solr.core.testCollection1.shard1.replica_n1'
> (registry 'solr.core.testCollection1.shard1.replica_n1') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>    [junit4]   2> 715160 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.m.r.SolrJmxReporter JMX
> monitoring for 'solr_33501.solr.core.testCollection1.shard2.replica_n3'
> (registry 'solr.core.testCollection1.shard2.replica_n3') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>    [junit4]   2> 715160 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore
> solr.RecoveryStrategy.Builder
>    [junit4]   2> 715161 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore
> solr.RecoveryStrategy.Builder
>    [junit4]   2> 715161 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore
> [[testCollection1_shard1_replica_n1] ] Opening new SolrCore at
> [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection1_shard1_replica_n1],
> dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection1_shard1_replica_n1/data/]
>    [junit4]   2> 715161 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore
> [[testCollection1_shard2_replica_n3] ] Opening new SolrCore at
> [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection1_shard2_replica_n3],
> dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection1_shard2_replica_n3/data/]
>    [junit4]   2> 715228 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateHandler Using UpdateLog
> implementation: org.apache.solr.update.UpdateLog
>    [junit4]   2> 715228 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateLog Initializing
> UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100
> maxNumLogsToKeep=10 numVersionBuckets=65536
>    [junit4]   2> 715230 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.u.CommitTracker Hard AutoCommit:
> disabled
>    [junit4]   2> 715230 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.u.CommitTracker Soft AutoCommit:
> disabled
>    [junit4]   2> 715233 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.s.SolrIndexSearcher Opening
> [Searcher@15532e2[testCollection1_shard2_replica_n3] main]
>    [junit4]   2> 715238 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.r.ManagedResourceStorage
> Configured ZooKeeperStorageIO with znodeBase: /configs/config
>    [junit4]   2> 715239 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog
> implementation: org.apache.solr.update.UpdateLog
>    [junit4]   2> 715239 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing
> UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100
> maxNumLogsToKeep=10 numVersionBuckets=65536
>    [junit4]   2> 715240 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.r.ManagedResourceStorage Loaded
> null at path _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/config
>    [junit4]   2> 715241 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit:
> disabled
>    [junit4]   2> 715241 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.h.ReplicationHandler Commits
> will be reserved for 10000ms.
>    [junit4]   2> 715241 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit:
> disabled
>    [junit4]   2> 715241 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateLog Could not find max
> version in index or recent updates, using new clock 1580641278248353792
>    [junit4]   2> 715243 INFO  (searcherExecutor-2416-thread-1-processing-n:
> 127.0.0.1:33501_solr x:testCollection1_shard2_replica_n3 s:shard2
> c:testCollection1 r:core_node4) [n:127.0.0.1:33501_solr c:testCollection1
> s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore
> [testCollection1_shard2_replica_n3] Registered new searcher Searcher@15532e2[testCollection1_shard2_replica_n3]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>    [junit4]   2> 715244 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening
> [Searcher@1202a80[testCollection1_shard1_replica_n1] main]
>    [junit4]   2> 715246 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage
> Configured ZooKeeperStorageIO with znodeBase: /configs/config
>    [junit4]   2> 715246 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded
> null at path _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/config
>    [junit4]   2> 715248 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits
> will be reserved for 10000ms.
>    [junit4]   2> 715249 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext
> Enough replicas found to continue.
>    [junit4]   2> 715249 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext I
> may be the new leader - try and sync
>    [junit4]   2> 715250 INFO  (searcherExecutor-2417-thread-1-processing-n:
> 127.0.0.1:33501_solr x:testCollection1_shard1_replica_n1 s:shard1
> c:testCollection1 r:core_node2) [n:127.0.0.1:33501_solr c:testCollection1
> s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore
> [testCollection1_shard1_replica_n1] Registered new searcher Searcher@1202a80[testCollection1_shard1_replica_n1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>    [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy Sync replicas to
> https://127.0.0.1:33501/solr/testCollection1_shard2_replica_n3/
>    [junit4]   2> 715250 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max
> version in index or recent updates, using new clock 1580641278257790976
>    [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy Sync Success -
> now sync replicas to me
>    [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy
> https://127.0.0.1:33501/solr/testCollection1_shard2_replica_n3/ has no
> replicas
>    [junit4]   2> 715250 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext
> Found all replicas participating in election, clear LIR
>    [junit4]   2> 715253 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext I
> am the new leader:
> https://127.0.0.1:33501/solr/testCollection1_shard2_replica_n3/ shard2
>    [junit4]   2> 715256 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext
> Enough replicas found to continue.
>    [junit4]   2> 715256 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I
> may be the new leader - try and sync
>    [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to
> https://127.0.0.1:33501/solr/testCollection1_shard1_replica_n1/
>    [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success -
> now sync replicas to me
>    [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy
> https://127.0.0.1:33501/solr/testCollection1_shard1_replica_n1/ has no
> replicas
>    [junit4]   2> 715257 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext
> Found all replicas participating in election, clear LIR
>    [junit4]   2> 715260 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I
> am the new leader:
> https://127.0.0.1:33501/solr/testCollection1_shard1_replica_n1/ shard1
>    [junit4]   2> 715363 INFO
> (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/testCollection1/state.json] for collection
> [testCollection1] has occurred - updating... (live nodes size: [1])
>    [junit4]   2> 715363 INFO
> (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/testCollection1/state.json] for collection
> [testCollection1] has occurred - updating... (live nodes size: [1])
>    [junit4]   2> 715405 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.c.ZkController I am the leader,
> no recovery necessary
>    [junit4]   2> 715406 INFO  (qtp20592558-5917) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard2 r:core_node4
> x:testCollection1_shard2_replica_n3] o.a.s.s.HttpSolrCall [admin]
> webapp=null path=/admin/cores
> params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection1_shard2_replica_n3&action=CREATE&numShards=2&collection=testCollection1&shard=shard2&wt=javabin&version=2&replicaType=NRT}
> status=0 QTime=1287
>    [junit4]   2> 715412 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.c.ZkController I am the leader,
> no recovery necessary
>    [junit4]   2> 715415 INFO  (qtp20592558-5918) [n:127.0.0.1:33501_solr
> c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin]
> webapp=null path=/admin/cores
> params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=config&newCollection=true&name=testCollection1_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection1&shard=shard1&wt=javabin&version=2&replicaType=NRT}
> status=0 QTime=1296
>    [junit4]   2> 715419 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>   ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for
> at most 30 seconds. Check all shard replicas
>    [junit4]   2> 715517 INFO
> (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/testCollection1/state.json] for collection
> [testCollection1] has occurred - updating... (live nodes size: [1])
>    [junit4]   2> 715517 INFO
> (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/testCollection1/state.json] for collection
> [testCollection1] has occurred - updating... (live nodes size: [1])
>    [junit4]   2> 715803 INFO
> (OverseerCollectionConfigSetProcessor-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path:
> /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor
> may have disconnected from ZooKeeper
>    [junit4]   2> 716419 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> params={replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2}
> status=0 QTime=2622
>    [junit4]   2> 716421 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr
>   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with
> params
> replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection2&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2
> and sendToOCPQueue=true
>    [junit4]   2> 716422 INFO
> (OverseerThreadFactory-2413-thread-2-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.CreateCollectionCmd Create
> collection testCollection2
>    [junit4]   2> 716627 INFO
> (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
>    [junit4]   2>   "operation":"ADDREPLICA",
>    [junit4]   2>   "collection":"testCollection2",
>    [junit4]   2>   "shard":"shard1",
>    [junit4]   2>   "core":"testCollection2_shard1_replica_n1",
>    [junit4]   2>   "state":"down",
>    [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
>    [junit4]   2>   "type":"NRT"}
>    [junit4]   2> 716629 INFO
> (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.o.SliceMutator createReplica() {
>    [junit4]   2>   "operation":"ADDREPLICA",
>    [junit4]   2>   "collection":"testCollection2",
>    [junit4]   2>   "shard":"shard2",
>    [junit4]   2>   "core":"testCollection2_shard2_replica_n2",
>    [junit4]   2>   "state":"down",
>    [junit4]   2>   "base_url":"https://127.0.0.1:33501/solr",
>    [junit4]   2>   "type":"NRT"}
>    [junit4]   2> 716831 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
>   ] o.a.s.h.a.CoreAdminOperation core create command
> qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection2_shard2_replica_n2&action=CREATE&numShards=2&collection=testCollection2&shard=shard2&wt=javabin&version=2&replicaType=NRT
>    [junit4]   2> 716833 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
>   ] o.a.s.h.a.CoreAdminOperation core create command
> qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=testCollection2_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection2&shard=shard1&wt=javabin&version=2&replicaType=NRT
>    [junit4]   2> 716937 INFO
> (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/testCollection2/state.json] for collection
> [testCollection2] has occurred - updating... (live nodes size: [1])
>    [junit4]   2> 716937 INFO
> (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/testCollection2/state.json] for collection
> [testCollection2] has occurred - updating... (live nodes size: [1])
>    [junit4]   2> 717844 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene
> MatchVersion: 7.1.0
>    [junit4]   2> 717844 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene
> MatchVersion: 7.1.0
>    [junit4]   2> 717870 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.s.IndexSchema
> [testCollection2_shard1_replica_n1] Schema name=minimal
>    [junit4]   2> 717870 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.s.IndexSchema
> [testCollection2_shard2_replica_n2] Schema name=minimal
>    [junit4]   2> 717874 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema
> minimal/1.1 with uniqueid field id
>    [junit4]   2> 717874 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema
> minimal/1.1 with uniqueid field id
>    [junit4]   2> 717874 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.c.CoreContainer Creating
> SolrCore 'testCollection2_shard1_replica_n1' using configuration from
> collection testCollection2, trusted=true
>    [junit4]   2> 717874 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.c.CoreContainer Creating
> SolrCore 'testCollection2_shard2_replica_n2' using configuration from
> collection testCollection2, trusted=true
>    [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX
> monitoring for 'solr_33501.solr.core.testCollection2.shard2.replica_n2'
> (registry 'solr.core.testCollection2.shard2.replica_n2') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>    [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX
> monitoring for 'solr_33501.solr.core.testCollection2.shard1.replica_n1'
> (registry 'solr.core.testCollection2.shard1.replica_n1') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@4525a2
>    [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore
> solr.RecoveryStrategy.Builder
>    [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore
> solr.RecoveryStrategy.Builder
>    [junit4]   2> 717875 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore
> [[testCollection2_shard2_replica_n2] ] Opening new SolrCore at
> [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection2_shard2_replica_n2],
> dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection2_shard2_replica_n2/data/]
>    [junit4]   2> 717875 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore
> [[testCollection2_shard1_replica_n1] ] Opening new SolrCore at
> [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/testCollection2_shard1_replica_n1],
> dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001/tempDir-001/node1/./testCollection2_shard1_replica_n1/data/]
>    [junit4]   2> 717939 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog
> implementation: org.apache.solr.update.UpdateLog
>    [junit4]   2> 717940 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateLog Initializing
> UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100
> maxNumLogsToKeep=10 numVersionBuckets=65536
>    [junit4]   2> 717941 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit:
> disabled
>    [junit4]   2> 717941 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit:
> disabled
>    [junit4]   2> 717942 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog
> implementation: org.apache.solr.update.UpdateLog
>    [junit4]   2> 717942 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening
> [Searcher@7b3d53[testCollection2_shard1_replica_n1] main]
>    [junit4]   2> 717942 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateLog Initializing
> UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100
> maxNumLogsToKeep=10 numVersionBuckets=65536
>    [junit4]   2> 717943 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage
> Configured ZooKeeperStorageIO with znodeBase: /configs/config
>    [junit4]   2> 717944 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded
> null at path _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/config
>    [junit4]   2> 717944 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit:
> disabled
>    [junit4]   2> 717944 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit:
> disabled
>    [junit4]   2> 717945 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits
> will be reserved for 10000ms.
>    [junit4]   2> 717946 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening
> [Searcher@17e3347[testCollection2_shard2_replica_n2] main]
>    [junit4]   2> 717946 INFO  (searcherExecutor-2426-thread-1-processing-n:
> 127.0.0.1:33501_solr x:testCollection2_shard1_replica_n1 s:shard1
> c:testCollection2 r:core_node3) [n:127.0.0.1:33501_solr c:testCollection2
> s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore
> [testCollection2_shard1_replica_n1] Registered new searcher Searcher@7b3d53[testCollection2_shard1_replica_n1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>    [junit4]   2> 717946 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max
> version in index or recent updates, using new clock 1580641281084751872
>    [junit4]   2> 717946 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.r.ManagedResourceStorage
> Configured ZooKeeperStorageIO with znodeBase: /configs/config
>    [junit4]   2> 717947 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded
> null at path _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/config
>    [junit4]   2> 717947 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits
> will be reserved for 10000ms.
>    [junit4]   2> 717948 INFO  (searcherExecutor-2427-thread-1-processing-n:
> 127.0.0.1:33501_solr x:testCollection2_shard2_replica_n2 s:shard2
> c:testCollection2 r:core_node4) [n:127.0.0.1:33501_solr c:testCollection2
> s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore
> [testCollection2_shard2_replica_n2] Registered new searcher Searcher@17e3347[testCollection2_shard2_replica_n2]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>    [junit4]   2> 717949 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max
> version in index or recent updates, using new clock 1580641281087897600
>    [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext
> Enough replicas found to continue.
>    [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I
> may be the new leader - try and sync
>    [junit4]   2> 717974 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to
> https://127.0.0.1:33501/solr/testCollection2_shard1_replica_n1/
>    [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success -
> now sync replicas to me
>    [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy
> https://127.0.0.1:33501/solr/testCollection2_shard1_replica_n1/ has no
> replicas
>    [junit4]   2> 717975 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext
> Found all replicas participating in election, clear LIR
>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext
> Enough replicas found to continue.
>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I
> may be the new leader - try and sync
>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to
> https://127.0.0.1:33501/solr/testCollection2_shard2_replica_n2/
>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success -
> now sync replicas to me
>    [junit4]   2> 717975 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy
> https://127.0.0.1:33501/solr/testCollection2_shard2_replica_n2/ has no
> replicas
>    [junit4]   2> 717976 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext
> Found all replicas participating in election, clear LIR
>    [junit4]   2> 717977 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I
> am the new leader:
> https://127.0.0.1:33501/solr/testCollection2_shard1_replica_n1/ shard1
>    [junit4]   2> 717978 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I
> am the new leader:
> https://127.0.0.1:33501/solr/testCollection2_shard2_replica_n2/ shard2
>    [junit4]   2> 718081 INFO
> (zkCallback-1001-thread-2-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/testCollection2/state.json] for collection
> [testCollection2] has occurred - updating... (live nodes size: [1])
>    [junit4]   2> 718082 INFO
> (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/testCollection2/state.json] for collection
> [testCollection2] has occurred - updating... (live nodes size: [1])
>    [junit4]   2> 718129 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.c.ZkController I am the leader,
> no recovery necessary
>    [junit4]   2> 718129 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.c.ZkController I am the leader,
> no recovery necessary
>    [junit4]   2> 718130 INFO  (qtp20592558-5948) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin]
> webapp=null path=/admin/cores
> params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=testCollection2_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection2&shard=shard1&wt=javabin&version=2&replicaType=NRT}
> status=0 QTime=1296
>    [junit4]   2> 718130 INFO  (qtp20592558-5947) [n:127.0.0.1:33501_solr
> c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin]
> webapp=null path=/admin/cores
> params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection2_shard2_replica_n2&action=CREATE&numShards=2&collection=testCollection2&shard=shard2&wt=javabin&version=2&replicaType=NRT}
> status=0 QTime=1299
>    [junit4]   2> 718133 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr
>   ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for
> at most 30 seconds. Check all shard replicas
>    [junit4]   2> 718234 INFO
> (zkCallback-1001-thread-1-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/testCollection2/state.json] for collection
> [testCollection2] has occurred - updating... (live nodes size: [1])
>    [junit4]   2> 718234 INFO
> (zkCallback-1001-thread-3-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/testCollection2/state.json] for collection
> [testCollection2] has occurred - updating... (live nodes size: [1])
>    [junit4]   2> 718423 INFO
> (OverseerCollectionConfigSetProcessor-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path:
> /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor
> may have disconnected from ZooKeeper
>    [junit4]   2> 719133 INFO  (qtp20592558-5911) [n:127.0.0.1:33501_solr
>   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> params={replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection2&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2}
> status=0 QTime=2712
>    [junit4]   2> 719134 INFO
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish -
> collection: testCollection1 failOnTimeout:true timeout (sec):30
>    [junit4]   2> 719134 INFO
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection:
> testCollection1
>    [junit4]   2> 719134 INFO
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish -
> collection: testCollection2 failOnTimeout:true timeout (sec):30
>    [junit4]   2> 719134 INFO
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection:
> testCollection2
>    [junit4]   2> 719145 INFO
> (TEST-LeaderElectionContextKeyTest.test-seed#[D426A0545BC5456]) [    ]
> o.a.s.SolrTestCaseJ4 ###Starting test
>    [junit4]   2> 719153 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>   ] o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.core.testCollection1.shard2.replica_n3, tag=null
>    [junit4]   2> 719154 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>   ] o.a.s.m.r.SolrJmxReporter Closing reporter
> [org.apache.solr.metrics.reporters.SolrJmxReporter@b5e43b: rootName =
> solr_33501, domain = solr.core.testCollection1.shard2.replica_n3, service
> url = null, agent id = null] for registry
> solr.core.testCollection1.shard2.replica_n3 /
> com.codahale.metrics.MetricRegistry@100889b
>    [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>   ] o.a.s.c.SolrCore [testCollection1_shard2_replica_n3]  CLOSING SolrCore
> org.apache.solr.core.SolrCore@8e18e5
>    [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>   ] o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.core.testCollection1.shard2.replica_n3, tag=9312485
>    [junit4]   2> 719174 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>   ] o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.collection.testCollection1.shard2.leader, tag=9312485
>    [junit4]   2> 719179 INFO  (qtp20592558-5916) [n:127.0.0.1:33501_solr
>   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> params={deleteInstanceDir=false&core=testCollection1_shard2_replica_n3&deleteDataDir=false&action=UNLOAD&indexInfo=true&wt=javabin&version=2&deleteIndex=false}
> status=0 QTime=26
>    [junit4]   2> 719181 INFO
> (TEST-LeaderElectionContextKeyTest.test-seed#[D426A0545BC5456]) [    ]
> o.a.s.SolrTestCaseJ4 ###Ending test
>    [junit4]   2> NOTE: reproduce with: ant test
> -Dtestcase=LeaderElectionContextKeyTest -Dtests.method=test
> -Dtests.seed=D426A0545BC5456 -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=mk -Dtests.timezone=Canada/Mountain -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
>    [junit4] FAILURE 0.05s J1 | LeaderElectionContextKeyTest.test <<<
>    [junit4]    > Throwable #1: java.lang.AssertionError
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([D426A0545BC5456:851655DFEB4039AE]:0)
>    [junit4]    >        at
> org.apache.solr.cloud.LeaderElectionContextKeyTest.test(LeaderElectionContextKeyTest.java:88)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:748)
>    [junit4]   2> 719191 INFO  (jetty-closer-995-thread-1) [    ]
> o.e.j.s.AbstractConnector Stopped ServerConnector@d0d870{SSL,[ssl,
> http/1.1]}{127.0.0.1:0}
>    [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ]
> o.a.s.c.CoreContainer Shutting down CoreContainer instance=31261178
>    [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ]
> o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node,
> tag=null
>    [junit4]   2> 719192 INFO  (jetty-closer-995-thread-1) [    ]
> o.a.s.m.r.SolrJmxReporter Closing reporter
> [org.apache.solr.metrics.reporters.SolrJmxReporter@cfcf8d: rootName =
> solr_33501, domain = solr.node, service url = null, agent id = null] for
> registry solr.node / com.codahale.metrics.MetricRegistry@8cc4d4
>    [junit4]   2> 719200 INFO  (jetty-closer-995-thread-1) [    ]
> o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm,
> tag=null
>    [junit4]   2> 719200 INFO  (jetty-closer-995-thread-1) [    ]
> o.a.s.m.r.SolrJmxReporter Closing reporter
> [org.apache.solr.metrics.reporters.SolrJmxReporter@18f30a8: rootName =
> solr_33501, domain = solr.jvm, service url = null, agent id = null] for
> registry solr.jvm / com.codahale.metrics.MetricRegistry@1bf26ed
>    [junit4]   2> 719208 INFO  (jetty-closer-995-thread-1) [    ]
> o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty,
> tag=null
>    [junit4]   2> 719208 INFO  (jetty-closer-995-thread-1) [    ]
> o.a.s.m.r.SolrJmxReporter Closing reporter
> [org.apache.solr.metrics.reporters.SolrJmxReporter@f110c8: rootName =
> solr_33501, domain = solr.jetty, service url = null, agent id = null] for
> registry solr.jetty / com.codahale.metrics.MetricRegistry@192a8b1
>    [junit4]   2> 719209 INFO  (jetty-closer-995-thread-1) [    ]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.cluster, tag=null
>    [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-1)
> [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore
> [testCollection1_shard1_replica_n1]  CLOSING SolrCore
> org.apache.solr.core.SolrCore@16aac00
>    [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-1)
> [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing
> metric reporters for registry=solr.core.testCollection1.shard1.replica_n1,
> tag=23768064
>    [junit4]   2> 719214 INFO  (coreCloseExecutor-2436-thread-2)
> [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore
> [testCollection2_shard1_replica_n1]  CLOSING SolrCore
> org.apache.solr.core.SolrCore@17509de
>    [junit4]   2> 719215 INFO  (coreCloseExecutor-2436-thread-3)
> [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore
> [testCollection2_shard2_replica_n2]  CLOSING SolrCore
> org.apache.solr.core.SolrCore@140f2a1
>    [junit4]   2> 719216 INFO  (coreCloseExecutor-2436-thread-1)
> [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing
> reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@169f449:
> rootName = solr_33501, domain =
> solr.core.testCollection1.shard1.replica_n1, service url = null, agent id =
> null] for registry solr.core.testCollection1.shard1.replica_n1 /
> com.codahale.metrics.MetricRegistry@121b656
>    [junit4]   2> 719230 INFO  (coreCloseExecutor-2436-thread-2)
> [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing
> metric reporters for registry=solr.core.testCollection2.shard1.replica_n1,
> tag=24447454
>    [junit4]   2> 719230 INFO  (coreCloseExecutor-2436-thread-2)
> [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing
> reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@e8b978:
> rootName = solr_33501, domain =
> solr.core.testCollection2.shard1.replica_n1, service url = null, agent id =
> null] for registry solr.core.testCollection2.shard1.replica_n1 /
> com.codahale.metrics.MetricRegistry@113466a
>    [junit4]   2> 719244 INFO  (coreCloseExecutor-2436-thread-3)
> [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing
> metric reporters for registry=solr.core.testCollection2.shard2.replica_n2,
> tag=21033633
>    [junit4]   2> 719244 INFO  (coreCloseExecutor-2436-thread-3)
> [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing
> reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@cfc71:
> rootName = solr_33501, domain =
> solr.core.testCollection2.shard2.replica_n2, service url = null, agent id =
> null] for registry solr.core.testCollection2.shard2.replica_n2 /
> com.codahale.metrics.MetricRegistry@13e256e
>    [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-1)
> [n:127.0.0.1:33501_solr c:testCollection1 s:shard1 r:core_node2
> x:testCollection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing
> metric reporters for
> registry=solr.collection.testCollection1.shard1.leader, tag=23768064
>    [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-2)
> [n:127.0.0.1:33501_solr c:testCollection2 s:shard1 r:core_node3
> x:testCollection2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing
> metric reporters for
> registry=solr.collection.testCollection2.shard1.leader, tag=24447454
>    [junit4]   2> 719257 INFO  (coreCloseExecutor-2436-thread-3)
> [n:127.0.0.1:33501_solr c:testCollection2 s:shard2 r:core_node4
> x:testCollection2_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing
> metric reporters for
> registry=solr.collection.testCollection2.shard2.leader, tag=21033633
>    [junit4]   2> 719267 INFO  (jetty-closer-995-thread-1) [    ]
> o.a.s.c.Overseer Overseer
> (id=98790079762989059-127.0.0.1:33501_solr-n_0000000000) closing
>    [junit4]   2> 719267 INFO
> (OverseerStateUpdate-98790079762989059-127.0.0.1:33501_solr-n_0000000000)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.Overseer Overseer Loop exiting :
> 127.0.0.1:33501_solr
>    [junit4]   2> 720768 WARN
> (zkCallback-1001-thread-3-processing-n:127.0.0.1:33501_solr)
> [n:127.0.0.1:33501_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch
> triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired
> for /live_nodes]
>    [junit4]   2> 720768 INFO  (jetty-closer-995-thread-1) [    ]
> o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@38e81f
> {/solr,null,UNAVAILABLE}
>    [junit4]   2> 720770 ERROR
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper
> server won't take any action on ERROR or SHUTDOWN server state changes
>    [junit4]   2> 720770 INFO
> (SUITE-LeaderElectionContextKeyTest-seed#[D426A0545BC5456]-worker) [    ]
> o.a.s.c.ZkTestServer connecting to 127.0.0.1:32955 32955
>    [junit4]   2> 725792 INFO  (Thread-1606) [    ] o.a.s.c.ZkTestServer
> connecting to 127.0.0.1:32955 32955
>    [junit4]   2> 725792 WARN  (Thread-1606) [    ] o.a.s.c.ZkTestServer
> Watch limit violations:
>    [junit4]   2> Maximum concurrent create/delete watches above limit:
>    [junit4]   2>
>    [junit4]   2>        2       /solr/aliases.json
>    [junit4]   2>
>    [junit4]   2> Maximum concurrent data watches above limit:
>    [junit4]   2>
>    [junit4]   2>        5
>  /solr/collections/testCollection2/state.json
>    [junit4]   2>        5
>  /solr/collections/testCollection1/state.json
>    [junit4]   2>        2       /solr/clusterstate.json
>    [junit4]   2>        2       /solr/clusterprops.json
>    [junit4]   2>
>    [junit4]   2> Maximum concurrent children watches above limit:
>    [junit4]   2>
>    [junit4]   2>        2       /solr/live_nodes
>    [junit4]   2>        2       /solr/collections
>    [junit4]   2>
>    [junit4]   2> NOTE: leaving temporary files on disk at:
> /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderElectionContextKeyTest_D426A0545BC5456-001
>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {},
> docValues:{}, maxPointsInLeafNode=1086, maxMBSortInHeap=5.302125034778907,
> sim=RandomSimilarity(queryNorm=true): {}, locale=mk,
> timezone=Canada/Mountain
>    [junit4]   2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation
> 1.8.0_144 (32-bit)/cpus=8,threads=1,free=134427200,total=326377472
>    [junit4]   2> NOTE: All tests run in this JVM: [BasicFunctionalityTest,
> TestTestInjection, TestConfigSetsAPIZkFailure, TestNRTOpen,
> TestPolicyCloud, CollectionsAPIAsyncDistributedZkTest,
> DistributedDebugComponentTest, TestFieldTypeCollectionResource,
> SearchHandlerTest, TestHighFrequencyDictionaryFactory,
> DistributedIntervalFacetingTest, TestReRankQParserPlugin,
> DistribJoinFromCollectionTest, TestRTGBase,
> BigEndianAscendingWordDeserializerTest, DocValuesMultiTest,
> SolrCloudExampleTest, TestDistribDocBasedVersion, CoreAdminHandlerTest,
> TestCloudSchemaless, TestManagedSynonymFilterFactory,
> TestEmbeddedSolrServerAdminHandler, TestDistributedGrouping,
> BufferStoreTest, TestMissingGroups, TestPushWriter, PrimUtilsTest,
> TestNumericTerms64, ShardRoutingCustomTest, TestSimpleTrackingShardHandler,
> AddSchemaFieldsUpdateProcessorFactoryTest,
> TestCollectionsAPIViaSolrCloudCluster, TestCloudSearcherWarming,
> TestHashPartitioner, ClassificationUpdateProcessorFactoryTest,
> TestSolrIndexConfig, RulesTest, JsonLoaderTest, SOLR749Test,
> TestHdfsBackupRestoreCore, BadComponentTest, TestBulkSchemaAPI,
> BooleanFieldTest, TestSkipOverseerOperations,
> AsyncCallRequestStatusResponseTest, TestUseDocValuesAsStored,
> TestConfigSetImmutable, TestGeoJSONResponseWriter, TestCollectionAPIs,
> TestPKIAuthenticationPlugin, UUIDUpdateProcessorFallbackTest,
> SystemInfoHandlerTest, ParsingFieldUpdateProcessorsTest, ExplicitHLLTest,
> TestBlendedInfixSuggestions, TestCloudNestedDocsSort, ZkCLITest,
> UninvertDocValuesMergePolicyTest, TestRandomRequestDistribution,
> LeaderElectionContextKeyTest]
>    [junit4] Completed [189/734 (1!)] on J1 in 12.52s, 1 test, 1 failure
> <<< FAILURES!
>
> [...truncated 44564 lines...]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org

-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

[JENKINS] Lucene-Solr-7.x-Linux (32bit/jdk1.8.0_144) - Build # 567 - Failure!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/567/
Java: 32bit/jdk1.8.0_144 -server -XX:+UseG1GC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth:     1) Thread[id=33593, name=jetty-launcher-6266-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)    2) Thread[id=33581, name=jetty-launcher-6266-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth: 
   1) Thread[id=33593, name=jetty-launcher-6266-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   2) Thread[id=33581, name=jetty-launcher-6266-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
	at __randomizedtesting.SeedInfo.seed([DED1E3E6C4199206]:0)




Build Log:
[...truncated 13971 lines...]
   [junit4] Suite: org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth
   [junit4]   2> 2872428 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[DED1E3E6C4199206]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_DED1E3E6C4199206-001/init-core-data-001
   [junit4]   2> 2872428 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[DED1E3E6C4199206]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2872429 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[DED1E3E6C4199206]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2872444 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[DED1E3E6C4199206]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_DED1E3E6C4199206-001/tempDir-001
   [junit4]   2> 2872445 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[DED1E3E6C4199206]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2872445 INFO  (Thread-6649) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2872445 INFO  (Thread-6649) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2872446 ERROR (Thread-6649) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2872545 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[DED1E3E6C4199206]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:40601
   [junit4]   2> 2872549 INFO  (jetty-launcher-6266-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2872549 INFO  (jetty-launcher-6266-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2872555 INFO  (jetty-launcher-6266-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7f5e09{/solr,null,AVAILABLE}
   [junit4]   2> 2872555 INFO  (jetty-launcher-6266-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@13eeebf{/solr,null,AVAILABLE}
   [junit4]   2> 2872556 INFO  (jetty-launcher-6266-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@25bf00{SSL,[ssl, http/1.1]}{127.0.0.1:36555}
   [junit4]   2> 2872556 INFO  (jetty-launcher-6266-thread-2) [    ] o.e.j.s.Server Started @2875111ms
   [junit4]   2> 2872556 INFO  (jetty-launcher-6266-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36555}
   [junit4]   2> 2872556 ERROR (jetty-launcher-6266-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2872556 INFO  (jetty-launcher-6266-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 2872556 INFO  (jetty-launcher-6266-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2872556 INFO  (jetty-launcher-6266-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2872556 INFO  (jetty-launcher-6266-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-08T03:07:23.325Z
   [junit4]   2> 2872561 INFO  (jetty-launcher-6266-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@175ba9e{SSL,[ssl, http/1.1]}{127.0.0.1:42931}
   [junit4]   2> 2872561 INFO  (jetty-launcher-6266-thread-1) [    ] o.e.j.s.Server Started @2875116ms
   [junit4]   2> 2872562 INFO  (jetty-launcher-6266-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42931}
   [junit4]   2> 2872562 ERROR (jetty-launcher-6266-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2872562 INFO  (jetty-launcher-6266-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 2872562 INFO  (jetty-launcher-6266-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2872562 INFO  (jetty-launcher-6266-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2872562 INFO  (jetty-launcher-6266-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-08T03:07:23.331Z
   [junit4]   2> 2872563 INFO  (jetty-launcher-6266-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2872568 INFO  (jetty-launcher-6266-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2872579 INFO  (jetty-launcher-6266-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40601/solr
   [junit4]   2> 2872580 INFO  (jetty-launcher-6266-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40601/solr
   [junit4]   2> 2872631 INFO  (jetty-launcher-6266-thread-1) [n:127.0.0.1:42931_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2872631 INFO  (jetty-launcher-6266-thread-1) [n:127.0.0.1:42931_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42931_solr
   [junit4]   2> 2872632 INFO  (jetty-launcher-6266-thread-1) [n:127.0.0.1:42931_solr    ] o.a.s.c.Overseer Overseer (id=98791066384138245-127.0.0.1:42931_solr-n_0000000000) starting
   [junit4]   2> 2872634 INFO  (jetty-launcher-6266-thread-2) [n:127.0.0.1:36555_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2872634 INFO  (jetty-launcher-6266-thread-2) [n:127.0.0.1:36555_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36555_solr
   [junit4]   2> 2872637 INFO  (zkCallback-6278-thread-1-processing-n:127.0.0.1:36555_solr) [n:127.0.0.1:36555_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2872637 INFO  (zkCallback-6277-thread-1-processing-n:127.0.0.1:42931_solr) [n:127.0.0.1:42931_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2872638 INFO  (jetty-launcher-6266-thread-1) [n:127.0.0.1:42931_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42931_solr
   [junit4]   2> 2872638 INFO  (zkCallback-6277-thread-1-processing-n:127.0.0.1:42931_solr) [n:127.0.0.1:42931_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2872641 INFO  (zkCallback-6278-thread-1-processing-n:127.0.0.1:36555_solr) [n:127.0.0.1:36555_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2872706 INFO  (jetty-launcher-6266-thread-1) [n:127.0.0.1:42931_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 2872707 INFO  (jetty-launcher-6266-thread-1) [n:127.0.0.1:42931_solr    ] o.a.s.s.HadoopAuthPlugin Params: {proxyuser.anyHostUsersGroup.hosts=*, proxyuser.wrongHost.hosts=1.1.1.1.1.1, proxyuser.bogusGroup.groups=__some_bogus_group, proxyuser.localHostAnyGroup.groups=*, cookie.domain=127.0.0.1, proxyuser.wrongHost.groups=*, type=simple, proxyuser.noHosts.groups=*, proxyuser.anyHostUsersGroup.groups=jenkins, proxyuser.noGroups.hosts=*, proxyuser.anyHostAnyUser.groups=*, delegation-token.token-kind=solr-dt, proxyuser.bogusGroup.hosts=*, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, zk-dt-secret-manager.enable=true, proxyuser.localHostAnyGroup.hosts=localhost.localdomain,localhost,127.0.0.1, proxyuser.anyHostAnyUser.hosts=*, signer.secret.provider=zookeeper}
   [junit4]   2> 2872707 INFO  (jetty-launcher-6266-thread-1) [n:127.0.0.1:42931_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 2872717 INFO  (jetty-launcher-6266-thread-1-EventThread) [n:127.0.0.1:42931_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 2872738 INFO  (jetty-launcher-6266-thread-1) [n:127.0.0.1:42931_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 2872769 INFO  (jetty-launcher-6266-thread-1) [n:127.0.0.1:42931_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42931.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5df0f0
   [junit4]   2> 2872775 INFO  (jetty-launcher-6266-thread-1) [n:127.0.0.1:42931_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42931.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5df0f0
   [junit4]   2> 2872775 INFO  (jetty-launcher-6266-thread-1) [n:127.0.0.1:42931_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42931.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5df0f0
   [junit4]   2> 2872776 INFO  (jetty-launcher-6266-thread-1) [n:127.0.0.1:42931_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_DED1E3E6C4199206-001/tempDir-001/node1/.
   [junit4]   2> 2872811 INFO  (jetty-launcher-6266-thread-2) [n:127.0.0.1:36555_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 2872812 INFO  (jetty-launcher-6266-thread-2) [n:127.0.0.1:36555_solr    ] o.a.s.s.HadoopAuthPlugin Params: {proxyuser.anyHostUsersGroup.hosts=*, proxyuser.wrongHost.hosts=1.1.1.1.1.1, proxyuser.bogusGroup.groups=__some_bogus_group, proxyuser.localHostAnyGroup.groups=*, cookie.domain=127.0.0.1, proxyuser.wrongHost.groups=*, type=simple, proxyuser.noHosts.groups=*, proxyuser.anyHostUsersGroup.groups=jenkins, proxyuser.noGroups.hosts=*, proxyuser.anyHostAnyUser.groups=*, delegation-token.token-kind=solr-dt, proxyuser.bogusGroup.hosts=*, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, zk-dt-secret-manager.enable=true, proxyuser.localHostAnyGroup.hosts=localhost.localdomain,localhost,127.0.0.1, proxyuser.anyHostAnyUser.hosts=*, signer.secret.provider=zookeeper}
   [junit4]   2> 2872812 INFO  (jetty-launcher-6266-thread-2) [n:127.0.0.1:36555_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 2872813 INFO  (jetty-launcher-6266-thread-2-EventThread) [n:127.0.0.1:36555_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 2872819 INFO  (jetty-launcher-6266-thread-2) [n:127.0.0.1:36555_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 2872848 INFO  (jetty-launcher-6266-thread-2) [n:127.0.0.1:36555_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36555.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5df0f0
   [junit4]   2> 2872856 INFO  (jetty-launcher-6266-thread-2) [n:127.0.0.1:36555_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36555.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5df0f0
   [junit4]   2> 2872856 INFO  (jetty-launcher-6266-thread-2) [n:127.0.0.1:36555_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36555.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5df0f0
   [junit4]   2> 2872858 INFO  (jetty-launcher-6266-thread-2) [n:127.0.0.1:36555_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_DED1E3E6C4199206-001/tempDir-001/node2/.
   [junit4]   2> 2872878 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[DED1E3E6C4199206]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2872879 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[DED1E3E6C4199206]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40601/solr ready
   [junit4]   2> 2872893 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigHosts-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigHosts
   [junit4]   2> 2872951 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigHosts-seed#[DED1E3E6C4199206]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=noHosts&p=noHosts&t=simple-dt&e=1507468043711&s=1jtjpoM3wbT+CpFwd5CF3P2Fbk8="; Domain=127.0.0.1; Expires= , 08-10 -2017 13:07:23 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 08-10 -2017 13:07:23 GMT
   [junit4]   2> 2872951 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigHosts-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigHosts
   [junit4]   2> 2872970 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyWrongHost-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyWrongHost
   [junit4]   2> 2887990 WARN  (jetty-launcher-6266-thread-1-SendThread(127.0.0.1:40601)) [n:127.0.0.1:42931_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15161ms for sessionid 0x15ef9f392cf0007
   [junit4]   2> 2887990 WARN  (jetty-launcher-6266-thread-2-SendThread(127.0.0.1:40601)) [n:127.0.0.1:36555_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15161ms for sessionid 0x15ef9f392cf0008
   [junit4]   2> 2887990 WARN  (jetty-launcher-6266-thread-1-SendThread(127.0.0.1:40601)) [n:127.0.0.1:42931_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15213ms for sessionid 0x15ef9f392cf0005
   [junit4]   2> 2887990 WARN  (jetty-launcher-6266-thread-2-SendThread(127.0.0.1:40601)) [n:127.0.0.1:36555_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15132ms for sessionid 0x15ef9f392cf0006
   [junit4]   2> 2887990 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ef9f392cf0007, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2887992 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ef9f392cf0008, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2887992 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ef9f392cf0006, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2887992 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ef9f392cf0005, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2887997 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyWrongHost-seed#[DED1E3E6C4199206]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=wrongHost&p=wrongHost&t=simple-dt&e=1507468043741&s=qWqOG8CUA8ZBz9wfOPJ+IKrN8b8="; Domain=127.0.0.1; Expires= , 08-10 -2017 13:07:23 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 08-10 -2017 13:07:23 GMT
   [junit4]   2> 2887997 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyWrongHost-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyWrongHost
   [junit4]   2> 2888015 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNullProxyUser-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNullProxyUser
   [junit4]   2> 2888018 INFO  (qtp16057907-33548) [    ] o.a.s.s.SolrDispatchFilter Error authenticating
   [junit4]   2> java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:195)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthFilter.doFilter(HadoopAuthFilter.java:110)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthPlugin.doAuthenticate(HadoopAuthPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2888018 WARN  (qtp16057907-33548) [    ] o.e.j.s.ServletHandler /solr/admin/collections
   [junit4]   2> org.apache.solr.common.SolrException: Error during request authentication, 
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:457)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:195)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthFilter.doFilter(HadoopAuthFilter.java:110)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthPlugin.doAuthenticate(HadoopAuthPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
   [junit4]   2> 	... 29 more
   [junit4]   2> 2888020 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNullProxyUser-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNullProxyUser
   [junit4]   2> 2888035 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidGroup-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidGroup
   [junit4]   2> 2888046 WARN  (qtp16057907-33546) [    ] o.a.h.s.ShellBasedUnixGroupsMapping got exception trying to get groups for user bar: id: ‘bar’: no such user
   [junit4]   2> id: ‘bar’: no such user
   [junit4]   2> 
   [junit4]   2> 2888047 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidGroup-seed#[DED1E3E6C4199206]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=bogusGroup&p=bogusGroup&t=simple-dt&e=1507468058806&s=52Fp2UwT8zNk4RPq/ZgXESCM+4o="; Domain=127.0.0.1; Expires= , 08-10 -2017 13:07:38 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 08-10 -2017 13:07:38 GMT
   [junit4]   2> 2888048 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidGroup-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidGroup
   [junit4]   2> 2888065 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigGroups-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigGroups
   [junit4]   2> 2888068 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigGroups-seed#[DED1E3E6C4199206]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=noGroups&p=noGroups&t=simple-dt&e=1507468058836&s=gtTIUM7UKOHo6AnK9c1Genj5rco="; Domain=127.0.0.1; Expires= , 08-10 -2017 13:07:38 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 08-10 -2017 13:07:38 GMT
   [junit4]   2> 2888069 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigGroups-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigGroups
   [junit4]   2> 2888086 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateHost-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateHost
   [junit4]   2> 2888089 INFO  (qtp16057907-33550) [n:127.0.0.1:42931_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&user.name=localHostAnyGroup&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2888089 INFO  (qtp16057907-33550) [n:127.0.0.1:42931_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&user.name=localHostAnyGroup&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2888090 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateHost-seed#[DED1E3E6C4199206]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=localHostAnyGroup&p=localHostAnyGroup&t=simple-dt&e=1507468058858&s=b40CIIFJtTb3EQz/+Ci5bmORQLg="; Domain=127.0.0.1; Expires= , 08-10 -2017 13:07:38 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 08-10 -2017 13:07:38 GMT
   [junit4]   2> 2888090 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateHost-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateHost
   [junit4]   2> 2888090 INFO  (jetty-launcher-6266-thread-2-EventThread) [n:127.0.0.1:36555_solr    ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
   [junit4]   2> 2888098 WARN  (zkCallback-6277-thread-1-processing-n:127.0.0.1:42931_solr) [n:127.0.0.1:42931_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@bc6ef0 name: ZooKeeperConnection Watcher:127.0.0.1:40601/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2888098 WARN  (zkCallback-6277-thread-1-processing-n:127.0.0.1:42931_solr) [n:127.0.0.1:42931_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2888101 WARN  (zkCallback-6278-thread-3-processing-n:127.0.0.1:36555_solr) [n:127.0.0.1:36555_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@11e815d name: ZooKeeperConnection Watcher:127.0.0.1:40601/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2888101 WARN  (zkCallback-6284-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@62e79d name: ZooKeeperConnection Watcher:127.0.0.1:40601/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2888101 WARN  (zkCallback-6284-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2888101 WARN  (zkCallback-6278-thread-3-processing-n:127.0.0.1:36555_solr) [n:127.0.0.1:36555_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2888115 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateAnyHostAnyUser-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateAnyHostAnyUser
   [junit4]   2> 2888117 INFO  (qtp16057907-33536) [n:127.0.0.1:42931_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&user.name=anyHostAnyUser&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2888117 INFO  (qtp16057907-33536) [n:127.0.0.1:42931_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&user.name=anyHostAnyUser&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2888117 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateAnyHostAnyUser-seed#[DED1E3E6C4199206]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=anyHostAnyUser&p=anyHostAnyUser&t=simple-dt&e=1507468058885&s=lvWIku1+1Rzqq+eTzA5yycOdxB4="; Domain=127.0.0.1; Expires= , 08-10 -2017 13:07:38 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 08-10 -2017 13:07:38 GMT
   [junit4]   2> 2888117 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateAnyHostAnyUser-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateAnyHostAnyUser
   [junit4] IGNOR/A 0.00s J1 | TestImpersonationWithHadoopAuth.testForwarding
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://issues.apache.org/jira/browse/HADOOP-9893))
   [junit4]   2> 2888134 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidProxyUser-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidProxyUser
   [junit4]   2> 2888137 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidProxyUser-seed#[DED1E3E6C4199206]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=bar&p=bar&t=simple-dt&e=1507468058905&s=Okomm0M53QGD8Kafppw53pZ3yY0="; Domain=127.0.0.1; Expires= , 08-10 -2017 13:07:38 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 08-10 -2017 13:07:38 GMT
   [junit4]   2> 2888137 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidProxyUser-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidProxyUser
   [junit4]   2> 2888150 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateGroup-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateGroup
   [junit4]   2> 2888162 INFO  (qtp16057907-33536) [n:127.0.0.1:42931_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=jenkins&action=LIST&user.name=anyHostUsersGroup&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2888162 INFO  (qtp16057907-33536) [n:127.0.0.1:42931_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=jenkins&action=LIST&user.name=anyHostUsersGroup&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2888163 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateGroup-seed#[DED1E3E6C4199206]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=anyHostUsersGroup&p=anyHostUsersGroup&t=simple-dt&e=1507468058921&s=L9PigCJEz+XfUL/s9ItnMEX4+SY="; Domain=127.0.0.1; Expires= , 08-10 -2017 13:07:38 GMT; Secure; HttpOnly". Invalid 'expires' attribute: , 08-10 -2017 13:07:38 GMT
   [junit4]   2> 2888163 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateGroup-seed#[DED1E3E6C4199206]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateGroup
   [junit4]   2> 2888562 INFO  (jetty-closer-6267-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@25bf00{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2888562 INFO  (jetty-closer-6267-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@175ba9e{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2888563 INFO  (jetty-closer-6267-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=25508360
   [junit4]   2> 2888564 INFO  (jetty-closer-6267-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=28369172
   [junit4]   2> 2888564 INFO  (jetty-closer-6267-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2888564 INFO  (jetty-closer-6267-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2888564 INFO  (jetty-closer-6267-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1cbe710: rootName = solr_36555, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@68c407
   [junit4]   2> 2888564 INFO  (jetty-closer-6267-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@878bb8: rootName = solr_42931, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@1090481
   [junit4]   2> 2888572 INFO  (jetty-closer-6267-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2888573 INFO  (jetty-closer-6267-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@18c6110: rootName = solr_36555, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@18e05f6
   [junit4]   2> 2888573 INFO  (jetty-closer-6267-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2888573 INFO  (jetty-closer-6267-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1da84b7: rootName = solr_42931, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@18e05f6
   [junit4]   2> 2888579 INFO  (jetty-closer-6267-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2888579 INFO  (jetty-closer-6267-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@36e4bb: rootName = solr_42931, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@dc8450
   [junit4]   2> 2888580 INFO  (jetty-closer-6267-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2888580 INFO  (jetty-closer-6267-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@b4877f: rootName = solr_36555, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@dc8450
   [junit4]   2> 2889119 INFO  (jetty-launcher-6266-thread-1-EventThread) [n:127.0.0.1:42931_solr    ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
   [junit4]   2> 2889851 INFO  (jetty-closer-6267-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 2889851 INFO  (jetty-closer-6267-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2889851 INFO  (jetty-closer-6267-thread-1) [    ] o.a.s.c.Overseer Overseer (id=98791066384138245-127.0.0.1:42931_solr-n_0000000000) closing
   [junit4]   2> 2889851 INFO  (OverseerStateUpdate-98791066384138245-127.0.0.1:42931_solr-n_0000000000) [n:127.0.0.1:42931_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:42931_solr
   [junit4]   2> 2889852 WARN  (zkCallback-6277-thread-3-processing-n:127.0.0.1:42931_solr) [n:127.0.0.1:42931_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2889852 ERROR (Thread[Thread-6666,5,TGRP-TestImpersonationWithHadoopAuth]) [n:127.0.0.1:42931_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 2889853 INFO  (jetty-closer-6267-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7f5e09{/solr,null,UNAVAILABLE}
   [junit4]   2> 2890051 INFO  (zkCallback-6278-thread-3-processing-n:127.0.0.1:36555_solr) [n:127.0.0.1:36555_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36555_solr
   [junit4]   2> 2890051 INFO  (zkCallback-6278-thread-1-processing-n:127.0.0.1:36555_solr) [n:127.0.0.1:36555_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 2890051 INFO  (jetty-closer-6267-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 2890051 INFO  (jetty-closer-6267-thread-2) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2890052 WARN  (zkCallback-6278-thread-1-processing-n:127.0.0.1:36555_solr) [n:127.0.0.1:36555_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2891552 ERROR (Thread[Thread-6669,5,TGRP-TestImpersonationWithHadoopAuth]) [n:127.0.0.1:36555_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 2891553 INFO  (jetty-closer-6267-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@13eeebf{/solr,null,UNAVAILABLE}
   [junit4]   2> 2891554 ERROR (SUITE-TestImpersonationWithHadoopAuth-seed#[DED1E3E6C4199206]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2891554 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[DED1E3E6C4199206]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40601 40601
   [junit4]   2> 2896578 INFO  (Thread-6649) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40601 40601
   [junit4]   2> 2896578 WARN  (Thread-6649) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/aliases.json
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
   [junit4]   2> 	3	/solr/clusterstate.json
   [junit4]   2> 	3	/solr/clusterprops.json
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 	3	/solr/collections
   [junit4]   2> 	3	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
   [junit4]   2> 
   [junit4]   2> 2896578 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[DED1E3E6C4199206]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Oct 08, 2017 3:07:47 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> Oct 08, 2017 3:08:07 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth: 
   [junit4]   2>    1) Thread[id=33593, name=jetty-launcher-6266-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2>         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2>         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2>    2) Thread[id=33581, name=jetty-launcher-6266-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2>         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2>         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Oct 08, 2017 3:08:07 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=33593, name=jetty-launcher-6266-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2>    2) Thread[id=33581, name=jetty-launcher-6266-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2> 2916727 ERROR (jetty-launcher-6266-thread-2-EventThread) [n:127.0.0.1:36555_solr    ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2> 	at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2> 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> 2916727 ERROR (jetty-launcher-6266-thread-1-EventThread) [n:127.0.0.1:42931_solr    ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2> 	at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2> 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Oct 08, 2017 3:08:07 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=false): {}, locale=ko, timezone=Pacific/Chuuk
   [junit4]   2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=1,free=217287056,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestSortByMinMaxFunction, SegmentsInfoRequestHandlerTest, MergeStrategyTest, DirectSolrConnectionTest, TestDocTermOrds, TestRTGBase, TestRandomRequestDistribution, AddSchemaFieldsUpdateProcessorFactoryTest, CoreAdminRequestStatusTest, DistributedQueryComponentOptimizationTest, DistributedFacetPivotLargeTest, TestDefaultStatsCache, TestMissingGroups, TestStressVersions, TestInitQParser, CollectionsAPIDistributedZkTest, ShardRoutingCustomTest, TestReRankQParserPlugin, StressHdfsTest, CacheHeaderTest, TestFieldCacheWithThreads, TestExactStatsCache, QueryParsingTest, TestPKIAuthenticationPlugin, SimpleMLTQParserTest, SolrCmdDistributorTest, FileBasedSpellCheckerTest, TestSolrCloudWithSecureImpersonation, RestartWhileUpdatingTest, LeaderInitiatedRecoveryOnCommitTest, TestSkipOverseerOperations, DeleteStatusTest, TestUseDocValuesAsStored, SolrJmxReporterTest, CoreAdminHandlerTest, TestSolrJ, CachingDirectoryFactoryTest, PeerSyncWithIndexFingerprintCachingTest, TestFastLRUCache, MoreLikeThisHandlerTest, LukeRequestHandlerTest, TestXIncludeConfig, TestEmbeddedSolrServerAdminHandler, TestSchemaResource, TestCloudSchemaless, ConfigSetsAPITest, DisMaxRequestHandlerTest, ResourceLoaderTest, DirectoryFactoryTest, DistribJoinFromCollectionTest, TestImplicitCoreProperties, TestHdfsCloudBackupRestore, TestQueryWrapperFilter, TestDynamicFieldResource, TestBM25SimilarityFactory, AnalysisAfterCoreReloadTest, TestBlendedInfixSuggestions, SpellCheckComponentTest, TestXmlQParser, HdfsNNFailoverTest, ZkControllerTest, TestCoreDiscovery, HighlighterTest, TestSubQueryTransformer, TestHashPartitioner, UninvertDocValuesMergePolicyTest, UpdateRequestProcessorFactoryTest, CloneFieldUpdateProcessorFactoryTest, JsonLoaderTest, TestHighFrequencyDictionaryFactory, TestObjectReleaseTracker, ZkStateWriterTest, TestMiniSolrCloudClusterSSL, V2ApiIntegrationTest, TestTestInjection, SOLR749Test, TestFiltering, TestSolrIndexConfig, TestDelegationWithHadoopAuth, TestStressLiveNodes, UUIDUpdateProcessorFallbackTest, TestSegmentSorting, TestMacros, DocValuesNotIndexedTest, TestLeaderInitiatedRecoveryThread, TestPointFields, TestNumericTerms64, TestConfigSetsAPIZkFailure, SchemaVersionSpecificBehaviorTest, DirectSolrSpellCheckerTest, NoCacheHeaderTest, DistributedTermsComponentTest, TestNumericRangeQuery64, ConvertedLegacyTest, DistributedMLTComponentTest, TestLegacyTerms, SliceStateTest, ShardRoutingTest, SampleTest, CdcrVersionReplicationTest, InfixSuggestersTest, ChaosMonkeySafeLeaderTest, SearchHandlerTest, AnalyticsQueryTest, OverseerModifyCollectionTest, DistributedFacetPivotSmallTest, TestPolicyCloud, TestCursorMarkWithoutUniqueKey, TestFieldTypeCollectionResource, IndexSchemaRuntimeFieldTest, TestPushWriter, ScriptEngineTest, TemplateUpdateProcessorTest, DateRangeFieldTest, TestSolrConfigHandlerCloud, ZkSolrClientTest, DOMUtilTest, TestComplexPhraseQParserPlugin, ZkCLITest, TestSearcherReuse, TestTolerantUpdateProcessorRandomCloud, HdfsChaosMonkeyNothingIsSafeTest, UnloadDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest, LeaderElectionIntegrationTest, ClusterStateUpdateTest, TestRandomFaceting, TestRandomDVFaceting, TestDistributedSearch, TestDistributedGrouping, TestFaceting, TestRecovery, TermVectorComponentDistributedTest, TestRealTimeGet, TestJoin, TestReload, TestRangeQuery, TestCoreContainer, TestSolr4Spatial, SolrCoreTest, QueryElevationComponentTest, PeerSyncTest, BadIndexSchemaTest, TestSort, TestFunctionQuery, BasicFunctionalityTest, SoftAutoCommitTest, SolrIndexSplitterTest, TestExtendedDismaxParser, SolrRequestParserTest, DocValuesTest, SuggesterTSTTest, SuggesterTest, SpatialFilterTest, SuggesterWFSTTest, TestCSVLoader, PolyFieldTest, TestUpdate, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, StatelessScriptUpdateProcessorFactoryTest, SortByFunctionTest, TestRemoteStreaming, DocValuesMultiTest, DistanceFunctionTest, SolrInfoBeanTest, DebugComponentTest, TestQueryTypes, PrimitiveFieldTypeTest, XmlUpdateRequestHandlerTest, RequestHandlersTest, TestValueSourceCache, PathHierarchyTokenizerFactoryTest, TermVectorComponentTest, TestIndexingPerformance, RequiredFieldsTest, TestSolrQueryParser, LoggingHandlerTest, RegexBoostProcessorTest, SolrPluginUtilsTest, IndexSchemaTest, TestCSVResponseWriter, TestAnalyzedSuggestions, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, CSVRequestHandlerTest, HighlighterConfigTest, TestQuerySenderListener, AlternateDirectoryTest, TestQuerySenderNoQuery, CopyFieldTest, SolrIndexConfigTest, BadComponentTest, TestStressRecovery, TestMergePolicyConfig, TestSolrDeletionPolicy2, MultiTermTest, TestDocSet, TestSearchPerf, NumericFieldsTest, TestElisionMultitermQuery, TestFuzzyAnalyzedSuggestions, TestPostingsSolrHighlighter, NotRequiredUniqueKeyTest, TestCharFilters, TestCodecSupport, EchoParamsTest, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestFastOutputStream, OpenExchangeRatesOrgProviderTest, ChaosMonkeyNothingIsSafeTest, TestEmbeddedSolrServerSchemaAPI, TestJettySolrRunner, ConnectionReuseTest, AsyncCallRequestStatusResponseTest, CdcrBootstrapTest, ConcurrentDeleteAndCreateCollectionTest, CustomCollectionTest, DeleteNodeTest, DistribCursorPagingTest, DistributedQueueTest, ForceLeaderTest, LeaderElectionContextKeyTest, LeaderFailureAfterFreshStartTest, MoveReplicaHDFSTest, MoveReplicaTest, OverseerRolesTest, OverseerStatusTest, OverseerTaskQueueTest, RemoteQueryErrorTest, RollingRestartTest, SSLMigrationTest, SaslZkACLProviderTest, SolrXmlInZkTest, TestCloudRecovery, TestCloudSearcherWarming, TestDistribDocBasedVersion, TestDownShardTolerantSearch, TestPrepRecovery, TestRequestForwarding, TestTlogReplica, InfoHandlerTest, MetricsHandlerTest, PropertiesRequestHandlerTest, SecurityConfHandlerTest, StatsReloadRaceTest, ReplicaListTransformerTest, SpatialHeatmapFacetsTest, TestTrackingShardHandlerFactory, TestNumericRangeQuery32, SolrMetricManagerTest, SolrMetricReporterTest, SolrGangliaReporterTest, SolrCloudReportersTest, TestMacroExpander, JSONWriterTest, TestBinaryResponseWriter, TestCustomDocTransformer, TestExportWriter, TestRetrieveFieldsOptimizer, TestSubQueryTransformerDistrib, BadCopyFieldTest, BooleanFieldTest, ChangedSchemaMergeTest, ManagedSchemaRoundRobinCloudTest, TestSchemalessBufferedUpdates, AnalyticsMergeStrategyTest, TestReloadDeadlock, TestSimpleQParserPlugin, TestSolr4Spatial2, TestStandardQParsers, TestTrieFacet, TestXmlQParserPlugin, TestJsonFacetRefinement, TestJsonFacets, TestScoreJoinQPScore, TestJsonRequest, CloudMLTQParserTest, BasicAuthStandaloneTest, PKIAuthenticationIntegrationTest, TestRuleBasedAuthorizationPlugin, TestImpersonationWithHadoopAuth]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestImpersonationWithHadoopAuth -Dtests.seed=DED1E3E6C4199206 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ko -Dtests.timezone=Pacific/Chuuk -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestImpersonationWithHadoopAuth (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth: 
   [junit4]    >    1) Thread[id=33593, name=jetty-launcher-6266-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]    >    2) Thread[id=33581, name=jetty-launcher-6266-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DED1E3E6C4199206]:0)
   [junit4] Completed [715/734 (1!)] on J1 in 44.31s, 10 tests, 1 error, 1 skipped <<< FAILURES!

[...truncated 63 lines...]
   [junit4] JVM J2: stdout was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/temp/junit4-J2-20171008_021928_1777033359349122648113.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /home/jenkins/workspace/Lucene-Solr-7.x-Linux/heapdumps/java_pid32353.hprof ...
   [junit4] Heap dump file created [312559523 bytes in 0.554 secs]
   [junit4] <<< JVM J2: EOF ----

[...truncated 8296 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/build.xml:826: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/build.xml:778: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid32353.hprof

Total time: 82 minutes 54 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any