You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2019/02/06 03:04:32 UTC
[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-11) - Build # 3509 -
Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3509/
Java: 64bit/jdk-11 -XX:-UseCompressedOops -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([F49C4FF42EEF100B:22436EAE18F20393]:0)
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at org.apache.solr.cloud.TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry(TestLeaderElectionZkExpiry.java:95)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:834)
Build Log:
[...truncated 14326 lines...]
[junit4] Suite: org.apache.solr.cloud.TestLeaderElectionZkExpiry
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestLeaderElectionZkExpiry_F49C4FF42EEF100B-001/init-core-data-001
[junit4] 2> 1313771 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 1313771 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
[junit4] 2> 1313771 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 1313773 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testLeaderElectionWithZkExpiry
[junit4] 2> 1313776 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4d0414f2, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 1313854 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
[junit4] 2> 1313867 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1313867 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 1313877 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d0414f2
[junit4] 2> 1313882 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d0414f2
[junit4] 2> 1313882 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d0414f2
[junit4] 2> 1313883 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestLeaderElectionZkExpiry_F49C4FF42EEF100B-001/testLeaderElectionWithZkExpiry-solr-001
[junit4] 2> 1313884 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1313884 INFO (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1313885 INFO (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1313985 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkTestServer start zk server on port:43509
[junit4] 2> 1313985 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:43509
[junit4] 2> 1313985 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 43509
[junit4] 2> 1313987 INFO (zkConnectionManagerCallback-8818-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1313989 INFO (zkConnectionManagerCallback-8820-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1313990 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1314017 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr
[junit4] 2> 1314017 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.Overseer Overseer (id=72060270075838466-dummy.host.com:8984_solr-n_0000000000) starting
[junit4] 2> 1314019 INFO (zkConnectionManagerCallback-8829-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1314021 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43509/solr ready
[junit4] 2> 1314021 INFO (OverseerStateUpdate-72060270075838466-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Starting to work on the main queue : dummy.host.com:8984_solr
[junit4] 2> 1314021 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr
[junit4] 2> 1314022 INFO (OverseerStateUpdate-72060270075838466-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1314022 INFO (zkCallback-8828-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1314023 INFO (zkCallback-8828-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 1314123 WARN (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 1314123 WARN (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 1315946 WARN (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]-SendThread(localhost.localdomain:43509)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100026f104a0002 has expired
[junit4] 2> 1315946 WARN (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
[junit4] 2> 1315946 WARN (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
[junit4] 2> 1315946 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72060270075838466-dummy.host.com:8984_solr-n_0000000000) closing
[junit4] 2> 1315946 WARN (OverseerStateUpdate-72060270075838466-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Solr cannot talk to ZK, exiting Overseer main queue loop
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1541) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getChildren$4(SolrZkClient.java:346) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:346) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.fetchZkChildren(ZkDistributedQueue.java:415) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.firstChild(ZkDistributedQueue.java:392) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:446) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:212) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1315947 INFO (OverseerStateUpdate-72060270075838466-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr
[junit4] 2> 1315948 WARN (OverseerExitThread) [ ] o.a.s.c.Overseer Error communicating with ZooKeeper
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1316047 WARN (zkCallback-8821-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 1316047 WARN (ScheduledTrigger-7314-thread-1) [ ] o.a.s.c.a.TriggerEventQueue Exception peeking queue of trigger .scheduled_maintenance
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.scheduled_maintenance
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1541) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getChildren$4(SolrZkClient.java:346) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:346) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.fetchZkChildren(ZkDistributedQueue.java:415) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.firstChild(ZkDistributedQueue.java:392) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.firstElement(ZkDistributedQueue.java:509) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.peek(ZkDistributedQueue.java:150) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.peekEvent(TriggerEventQueue.java:69) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:611) [java/:?]
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
[junit4] 2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1316047 ERROR (OverseerAutoScalingTriggerThread-72060270075838466-dummy.host.com:8984_solr-n_0000000000) [ ] o.a.s.c.a.ScheduledTriggers Failed to add trigger .auto_add_replicas
[junit4] 2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:129) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:119) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:115) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:111) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueueFactory.makeQueue(ZkDistributedQueueFactory.java:36) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:49) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.<init>(ScheduledTriggers.java:566) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:220) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:245) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:108) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:127) ~[java/:?]
[junit4] 2> ... 9 more
[junit4] 2> 1316048 WARN (OverseerAutoScalingTriggerThread-72060270075838466-dummy.host.com:8984_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread Error removing old markers
[junit4] 2> org.apache.solr.common.AlreadyClosedException: null
[junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:269) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:262) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1316048 WARN (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
[junit4] 2> 1316049 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1316049 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
[junit4] 2> 1316049 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
[junit4] 2> 1316050 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 1316051 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72060270075838466-dummy.host.com:8984_solr-n_0000000000) closing
[junit4] 2> 1316052 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72060270075838466-dummy.host.com:8984_solr-n_0000000000) closing
[junit4] 2> 1316052 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72060270075838466-dummy.host.com:8984_solr-n_0000000000) closing
[junit4] 2> 1316052 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72060270075838466-dummy.host.com:8984_solr-n_0000000000) closing
[junit4] 2> 1316052 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr
[junit4] 2> 1316052 INFO (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=72060270075838468-dummy.host.com:8984_solr-n_0000000001) starting
[junit4] 2> 1316054 INFO (OverseerStateUpdate-72060270075838468-dummy.host.com:8984_solr-n_0000000001) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Starting to work on the main queue : dummy.host.com:8984_solr
[junit4] 2> 1316054 INFO (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Publish node=dummy.host.com:8984_solr as DOWN
[junit4] 2> 1316055 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn CancelledKeyException causing close of session 0x100026f104a0004
[junit4] 2> 1316155 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 1316155 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 1317409 WARN (zkConnectionManagerCallback-8822-thread-1-SendThread(localhost.localdomain:43509)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100026f104a0004 has expired
[junit4] 2> 1317409 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
[junit4] 2> 1317409 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
[junit4] 2> 1317409 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838468-dummy.host.com:8984_solr-n_0000000001) closing
[junit4] 2> 1317409 INFO (OverseerStateUpdate-72060270075838468-dummy.host.com:8984_solr-n_0000000001) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr
[junit4] 2> 1317420 WARN (OverseerExitThread) [ ] o.a.s.c.Overseer Error communicating with ZooKeeper
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1317655 WARN (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Could not publish node as down: KeeperErrorCode = Session expired for /overseer/queue/qn-
[junit4] 2> 1317655 INFO (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr
[junit4] 2> 1317655 ERROR (OverseerAutoScalingTriggerThread-72060270075838468-dummy.host.com:8984_solr-n_0000000001) [ ] o.a.s.c.a.ScheduledTriggers Failed to add trigger .scheduled_maintenance
[junit4] 2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.scheduled_maintenance
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:129) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:119) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:115) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:111) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueueFactory.makeQueue(ZkDistributedQueueFactory.java:36) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:49) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.<init>(ScheduledTriggers.java:566) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:220) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:245) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.scheduled_maintenance
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:108) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:127) ~[java/:?]
[junit4] 2> ... 9 more
[junit4] 2> 1317655 ERROR (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getAutoScalingConfig(ZkStateReader.java:215)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getAutoScalingConfig(ZkStateReader.java:201)
[junit4] 2> at org.apache.solr.cloud.ZkController.createEphemeralLiveNode(ZkController.java:1082)
[junit4] 2> at org.apache.solr.cloud.ZkController.access$500(ZkController.java:149)
[junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:376)
[junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:177)
[junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:58)
[junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:151)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:285)
[junit4] 2> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[junit4] 2> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
[junit4] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[junit4] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:834)
[junit4] 2>
[junit4] 2> 1317655 ERROR (OverseerAutoScalingTriggerThread-72060270075838468-dummy.host.com:8984_solr-n_0000000001) [ ] o.a.s.c.a.ScheduledTriggers Failed to add trigger .auto_add_replicas
[junit4] 2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:129) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:119) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:115) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:111) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueueFactory.makeQueue(ZkDistributedQueueFactory.java:36) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:49) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.<init>(ScheduledTriggers.java:566) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:220) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:245) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:108) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:127) ~[java/:?]
[junit4] 2> ... 9 more
[junit4] 2> 1317656 ERROR (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException:
[junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:428)
[junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:177)
[junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:58)
[junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:151)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:285)
[junit4] 2> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[junit4] 2> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
[junit4] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[junit4] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:834)
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getAutoScalingConfig(ZkStateReader.java:215)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getAutoScalingConfig(ZkStateReader.java:201)
[junit4] 2> at org.apache.solr.cloud.ZkController.createEphemeralLiveNode(ZkController.java:1082)
[junit4] 2> at org.apache.solr.cloud.ZkController.access$500(ZkController.java:149)
[junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:376)
[junit4] 2> ... 10 more
[junit4] 2>
[junit4] 2> 1317656 WARN (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
[junit4] 2> 1317656 INFO (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient Connected:false
[junit4] 2> 1317656 WARN (OverseerAutoScalingTriggerThread-72060270075838468-dummy.host.com:8984_solr-n_0000000001) [ ] o.a.s.c.a.OverseerTriggerThread Error removing old markers
[junit4] 2> org.apache.solr.common.AlreadyClosedException: null
[junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:269) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:261) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1317656 WARN (OverseerAutoScalingTriggerThread-72060270075838468-dummy.host.com:8984_solr-n_0000000001) [ ] o.a.s.c.a.OverseerTriggerThread Error removing old markers
[junit4] 2> org.apache.solr.common.AlreadyClosedException: null
[junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:269) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:262) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1317656 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
[junit4] 2> 1317657 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1317657 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
[junit4] 2> 1317657 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
[junit4] 2> 1317659 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838468-dummy.host.com:8984_solr-n_0000000001) closing
[junit4] 2> 1317659 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838468-dummy.host.com:8984_solr-n_0000000001) closing
[junit4] 2> 1317659 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838468-dummy.host.com:8984_solr-n_0000000001) closing
[junit4] 2> 1317659 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838468-dummy.host.com:8984_solr-n_0000000001) closing
[junit4] 2> 1317659 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr
[junit4] 2> 1317660 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=72060270075838469-dummy.host.com:8984_solr-n_0000000002) starting
[junit4] 2> 1317661 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Publish node=dummy.host.com:8984_solr as DOWN
[junit4] 2> 1317661 INFO (OverseerStateUpdate-72060270075838469-dummy.host.com:8984_solr-n_0000000002) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Starting to work on the main queue : dummy.host.com:8984_solr
[junit4] 2> 1317661 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr
[junit4] 2> 1317662 INFO (zkCallback-8821-thread-5) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1317662 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
[junit4] 2> 1317662 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient Connected:true
[junit4] 2> 1317763 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 1317763 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 1319246 WARN (zkConnectionManagerCallback-8822-thread-1-SendThread(localhost.localdomain:43509)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100026f104a0005 has expired
[junit4] 2> 1319246 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
[junit4] 2> 1319246 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
[junit4] 2> 1319246 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838469-dummy.host.com:8984_solr-n_0000000002) closing
[junit4] 2> 1319247 INFO (OverseerStateUpdate-72060270075838469-dummy.host.com:8984_solr-n_0000000002) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr
[junit4] 2> 1319247 ERROR (OverseerCollectionConfigSetProcessor-72060270075838469-dummy.host.com:8984_solr-n_0000000002) [n:dummy.host.com:8984_solr ] o.a.s.c.OverseerTaskProcessor Unable to prioritize overseer
[junit4] 2> java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method) ~[?:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:126) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:81) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.OverseerNodePrioritizer.prioritizeOverseerNodes(OverseerNodePrioritizer.java:68) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:187) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1319247 WARN (OverseerExitThread) [ ] o.a.s.c.Overseer Error communicating with ZooKeeper
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1319263 WARN (zkCallback-8821-thread-6) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 1319263 ERROR (OverseerAutoScalingTriggerThread-72060270075838469-dummy.host.com:8984_solr-n_0000000002) [ ] o.a.s.c.a.ScheduledTriggers Failed to add trigger .scheduled_maintenance
[junit4] 2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.scheduled_maintenance
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:129) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:119) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:115) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:111) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueueFactory.makeQueue(ZkDistributedQueueFactory.java:36) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:49) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.<init>(ScheduledTriggers.java:566) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:220) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:245) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.scheduled_maintenance
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:108) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:127) ~[java/:?]
[junit4] 2> ... 9 more
[junit4] 2> 1319264 ERROR (OverseerAutoScalingTriggerThread-72060270075838469-dummy.host.com:8984_solr-n_0000000002) [ ] o.a.s.c.a.ScheduledTriggers Failed to add trigger .auto_add_replicas
[junit4] 2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:129) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:119) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:115) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:111) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueueFactory.makeQueue(ZkDistributedQueueFactory.java:36) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:49) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.<init>(ScheduledTriggers.java:566) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:220) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:245) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:108) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:127) ~[java/:?]
[junit4] 2> ... 9 more
[junit4] 2> 1319264 WARN (OverseerAutoScalingTriggerThread-72060270075838469-dummy.host.com:8984_solr-n_0000000002) [ ] o.a.s.c.a.OverseerTriggerThread Error removing old markers
[junit4] 2> org.apache.solr.common.AlreadyClosedException: null
[junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:269) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:261) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1319264 WARN (OverseerAutoScalingTriggerThread-72060270075838469-dummy.host.com:8984_solr-n_0000000002) [ ] o.a.s.c.a.OverseerTriggerThread Error removing old markers
[junit4] 2> org.apache.solr.common.AlreadyClosedException: null
[junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:269) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:262) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1319265 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
[junit4] 2> 1319265 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1319265 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
[junit4] 2> 1319265 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
[junit4] 2> 1319266 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 1319267 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838469-dummy.host.com:8984_solr-n_0000000002) closing
[junit4] 2> 1319267 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838469-dummy.host.com:8984_solr-n_0000000002) closing
[junit4] 2> 1319267 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838469-dummy.host.com:8984_solr-n_0000000002) closing
[junit4] 2> 1319267 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838469-dummy.host.com:8984_solr-n_0000000002) closing
[junit4] 2> 1319267 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr
[junit4] 2> 1319267 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) starting
[junit4] 2> 1319269 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Publish node=dummy.host.com:8984_solr as DOWN
[junit4] 2> 1319269 INFO (OverseerStateUpdate-72060270075838470-dummy.host.com:8984_solr-n_0000000003) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Starting to work on the main queue : dummy.host.com:8984_solr
[junit4] 2> 1319269 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr
[junit4] 2> 1319269 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
[junit4] 2> 1319269 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient Connected:true
[junit4] 2> 1319270 INFO (zkCallback-8828-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1319275 INFO (zkCallback-8821-thread-6) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1319275 INFO (zkCallback-8828-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 1319375 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 1319376 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 1320563 WARN (zkConnectionManagerCallback-8822-thread-1-SendThread(localhost.localdomain:43509)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100026f104a0006 has expired
[junit4] 2> 1320563 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
[junit4] 2> 1320563 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
[junit4] 2> 1320563 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing
[junit4] 2> 1320563 WARN (OverseerCollectionConfigSetProcessor-72060270075838470-dummy.host.com:8984_solr-n_0000000003) [n:dummy.host.com:8984_solr ] o.a.s.c.OverseerTaskProcessor Overseer cannot talk to ZK
[junit4] 2> 1320563 INFO (OverseerStateUpdate-72060270075838470-dummy.host.com:8984_solr-n_0000000003) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr
[junit4] 2> 1320564 WARN (OverseerExitThread) [ ] o.a.s.c.Overseer Error communicating with ZooKeeper
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1320875 WARN (zkCallback-8821-thread-6) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 1320875 WARN (ScheduledTrigger-7326-thread-1) [ ] o.a.s.c.a.TriggerEventQueue Exception peeking queue of trigger .scheduled_maintenance
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.scheduled_maintenance
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1541) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getChildren$4(SolrZkClient.java:346) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:346) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.fetchZkChildren(ZkDistributedQueue.java:415) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.firstChild(ZkDistributedQueue.java:392) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.firstElement(ZkDistributedQueue.java:509) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.peek(ZkDistributedQueue.java:150) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.peekEvent(TriggerEventQueue.java:69) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:611) [java/:?]
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
[junit4] 2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1320875 ERROR (OverseerAutoScalingTriggerThread-72060270075838470-dummy.host.com:8984_solr-n_0000000003) [ ] o.a.s.c.a.ScheduledTriggers Failed to add trigger .auto_add_replicas
[junit4] 2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:129) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:119) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:115) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:111) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueueFactory.makeQueue(ZkDistributedQueueFactory.java:36) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:49) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.<init>(ScheduledTriggers.java:566) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:220) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:245) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:108) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:127) ~[java/:?]
[junit4] 2> ... 9 more
[junit4] 2> 1320877 WARN (OverseerAutoScalingTriggerThread-72060270075838470-dummy.host.com:8984_solr-n_0000000003) [ ] o.a.s.c.a.OverseerTriggerThread Error removing old markers
[junit4] 2> org.apache.solr.common.AlreadyClosedException: null
[junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:269) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:261) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1320877 WARN (OverseerAutoScalingTriggerThread-72060270075838470-dummy.host.com:8984_solr-n_0000000003) [ ] o.a.s.c.a.OverseerTriggerThread Error removing old markers
[junit4] 2> org.apache.solr.common.AlreadyClosedException: null
[junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:269) [java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:262) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1320877 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
[junit4] 2> 1320878 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1320878 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
[junit4] 2> 1320878 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
[junit4] 2> 1320878 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 1320983 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 1320983 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 1322315 WARN (zkConnectionManagerCallback-8822-thread-1-SendThread(localhost.localdomain:43509)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100026f104a0007 has expired
[junit4] 2> 1322315 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
[junit4] 2> 1322315 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
[junit4] 2> 1322315 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing
[junit4] 2> 1322315 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
[junit4] 2> 1322315 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1322315 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
[junit4] 2> 1322315 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
[junit4] 2> 1322424 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 1322424 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 1323900 WARN (zkConnectionManagerCallback-8822-thread-1-SendThread(localhost.localdomain:43509)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100026f104a0008 has expired
[junit4] 2> 1323900 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
[junit4] 2> 1323900 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
[junit4] 2> 1323900 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing
[junit4] 2> 1323900 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
[junit4] 2> 1323901 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1323901 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
[junit4] 2> 1323901 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
[junit4] 2> 1324001 ERROR (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /security.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$2(SolrZkClient.java:322)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:322)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.addSecurityNodeWatcher(ZkStateReader.java:485)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:468)
[junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:343)
[junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:177)
[junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:58)
[junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:151)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:533)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508)
[junit4] 2>
[junit4] 2> 1324001 ERROR (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException:
[junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:428)
[junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:177)
[junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:58)
[junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:151)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:533)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508)
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /security.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$2(SolrZkClient.java:322)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:322)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.addSecurityNodeWatcher(ZkStateReader.java:485)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:468)
[junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:343)
[junit4] 2> ... 5 more
[junit4] 2>
[junit4] 2> 1324001 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
[junit4] 2> 1324001 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient Connected:true
[junit4] 2> 1324002 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 1324002 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 1324024 INFO (zkConnectionManagerCallback-8831-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1325098 WARN (zkConnectionManagerCallback-8822-thread-1-SendThread(localhost.localdomain:43509)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100026f104a0009 has expired
[junit4] 2> 1325098 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
[junit4] 2> 1325098 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
[junit4] 2> 1325098 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing
[junit4] 2> 1325098 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
[junit4] 2> 1325099 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1325099 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
[junit4] 2> 1325099 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
[junit4] 2> 1325504 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing
[junit4] 2> 1325504 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing
[junit4] 2> 1325504 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing
[junit4] 2> 1325504 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing
[junit4] 2> 1325504 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr
[junit4] 2> 1325505 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) starting
[junit4] 2> 1325506 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing
[junit4] 2> 1325506 INFO (OverseerStateUpdate-72060270075838475-dummy.host.com:8984_solr-n_0000000004) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Starting to work on the main queue : dummy.host.com:8984_solr
[junit4] 2> 1325506 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Publish node=dummy.host.com:8984_solr as DOWN
[junit4] 2> 1325506 INFO (OverseerStateUpdate-72060270075838475-dummy.host.com:8984_solr-n_0000000004) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr
[junit4] 2> 1325508 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr
[junit4] 2> 1325508 ERROR (OverseerCollectionConfigSetProcessor-72060270075838475-dummy.host.com:8984_solr-n_0000000004) [n:dummy.host.com:8984_solr ] o.a.s.c.OverseerTaskProcessor Unable to prioritize overseer
[junit4] 2> java.lang.InterruptedException: null
[junit4] 2> at java.lang.Object.wait(Native Method) ~[?:?]
[junit4] 2> at java.lang.Object.wait(Object.java:328) ~[?:?]
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1411) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1106) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.OverseerNodePrioritizer.prioritizeOverseerNodes(OverseerNodePrioritizer.java:68) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:187) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?]
[junit4] 2> 1325508 WARN (OverseerExitThread) [ ] o.a.s.c.Overseer I (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) am exiting, but I'm still the leader
[junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing
[junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing
[junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing
[junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing
[junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing
[junit4] 2> 1325509 INFO (OverseerExitThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing
[junit4] 2> 1325509 INFO (OverseerExitThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing
[junit4] 2> 1325509 INFO (OverseerExitThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing
[junit4] 2> 1325509 INFO (zkCallback-8828-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1325509 INFO (zkCallback-8821-thread-5) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
[junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing
[junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient Connected:true
[junit4] 2> 1325509 INFO (OverseerExitThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing
[junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing
[junit4] 2> 1325509 WARN (OverseerExitThread) [ ] o.a.s.c.LeaderElector Deleting duplicate registration: /overseer_elect/election/72060270075838475-dummy.host.com:8984_solr-n_0000000005
[junit4] 2> 1325509 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.LeaderElector Deleting duplicate registration: /overseer_elect/election/72060270075838475-dummy.host.com:8984_solr-n_0000000006
[junit4] 2> 1325509 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.LeaderElector Deleting duplicate registration: /overseer_elect/election/72060270075838475-dummy.host.com:8984_solr-n_0000000005
[junit4] 2> 1325509 WARN (OverseerExitThread) [ ] o.a.s.c.LeaderElector Deleting duplicate registration: /overseer_elect/election/72060270075838475-dummy.host.com:8984_solr-n_0000000006
[junit4] 2> 1325509 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.LeaderElector Deleting duplicate registration: /overseer_elect/election/72060270075838475-dummy.host.com:8984_solr-n_0000000007
[junit4] 2> 1325509 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.LeaderElector Deleting duplicate registration: /overseer_elect/election/72060270075838475-dummy.host.com:8984_solr-n_0000000007
[junit4] 2> 1325510 INFO (OverseerExitThread) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr
[junit4] 2> 1325510 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr
[junit4] 2> 1325510 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr
[junit4] 2> 1325510 INFO (OverseerExitThread) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000007) starting
[junit4] 2> 1325510 ERROR (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.LeaderElector node exists
[junit4] 2> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:122) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
[junit4]
[...truncated too long message...]
] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 1384033 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3e2e79f7: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@3ededc45
[junit4] 2> 1384034 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkTestServer Shutting down ZkTestServer.
[junit4] 2> 1384035 WARN (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 9 /solr/aliases.json
[junit4] 2> 9 /solr/clusterprops.json
[junit4] 2> 7 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 9 /solr/clusterstate.json
[junit4] 2> 4 /solr/autoscaling.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 9 /solr/collections
[junit4] 2> 8 /solr/live_nodes
[junit4] 2> 3 /solr/overseer/queue
[junit4] 2> 3 /solr/overseer/queue-work
[junit4] 2> 2 /solr/overseer/collection-queue-work
[junit4] 2>
[junit4] 2> 1384037 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:43509
[junit4] 2> 1384037 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 43509
[junit4] 2> 1384037 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testLeaderElectionWithZkExpiry
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestLeaderElectionZkExpiry -Dtests.method=testLeaderElectionWithZkExpiry -Dtests.seed=F49C4FF42EEF100B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=dsb -Dtests.timezone=America/Asuncion -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 70.3s J1 | TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry <<<
[junit4] > Throwable #1: java.lang.AssertionError
[junit4] > at __randomizedtesting.SeedInfo.seed([F49C4FF42EEF100B:22436EAE18F20393]:0)
[junit4] > at org.apache.solr.cloud.TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry(TestLeaderElectionZkExpiry.java:95)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:834)
[junit4] 2> 1384040 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> 1384040 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=922172301
[junit4] 2> 1384040 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 1384040 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 1384040 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestLeaderElectionZkExpiry_F49C4FF42EEF100B-001
[junit4] 2> Feb 06, 2019 2:42:05 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 5 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=true): {}, locale=dsb, timezone=America/Asuncion
[junit4] 2> NOTE: Linux 4.15.0-45-generic amd64/Oracle Corporation 11 (64-bit)/cpus=8,threads=1,free=245205472,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [MetricsHistoryHandlerTest, RestartWhileUpdatingTest, TestTestInjection, TestReversedWildcardFilterFactory, DeleteStatusTest, TestTrieFacet, MultiSolrCloudTestCaseTest, TestSolrCoreParser, UtilsToolTest, BasicFunctionalityTest, TestSimLargeCluster, TestManagedResourceStorage, CollectionStateFormat2Test, TestCloudPivotFacet, SubstringBytesRefFilterTest, TestExpandComponent, SuggestComponentContextFilterQueryTest, TestFuzzyAnalyzedSuggestions, TestLRUCache, TestIndexSearcher, TestDocTermOrdsUninvertLimit, TestSearchPerf, OutOfBoxZkACLAndCredentialsProvidersTest, CSVRequestHandlerTest, URLClassifyProcessorTest, TestManagedSchemaAPI, TestMissingGroups, TestCloudNestedDocsSort, TestPartialUpdateDeduplication, SegmentsInfoRequestHandlerTest, JsonLoaderTest, TestBinaryResponseWriter, MergeStrategyTest, SolrCloudExampleTest, BadComponentTest, BasicAuthIntegrationTest, TestTolerantSearch, TestReRankQParserPlugin, BooleanFieldTest, TestShortCircuitedRequests, ZkSolrClientTest, DistributedUpdateProcessorTest, TestCloudDeleteByQuery, TestCollationFieldDocValues, SimpleMLTQParserTest, TestUniqueKeyFieldResource, TestMacros, TestRecoveryHdfs, SharedFSAutoReplicaFailoverTest, TestOverriddenPrefixQueryForCustomFieldType, AdminHandlersProxyTest, TransactionLogTest, TestJsonFacetsWithNestedObjects, SpellingQueryConverterTest, TermVectorComponentTest, TestPerFieldSimilarity, TriggerIntegrationTest, BlockJoinFacetRandomTest, TemplateUpdateProcessorTest, TestBlobHandler, TestPrepRecovery, SolrGraphiteReporterTest, IndexSchemaTest, TestCopyFieldCollectionResource, DateMathParserTest, TestSolrDeletionPolicy1, TestRandomFaceting, TestCorePropertiesReload, DistributedFacetExistsSmallTest, TestPolicyCloud, HttpPartitionOnCommitTest, LeaderTragicEventTest, ResponseHeaderTest, SpatialHeatmapFacetsTest, LukeRequestHandlerTest, DistributedQueryElevationComponentTest, DirectSolrSpellCheckerTest, TestSolrXml, TestSubQueryTransformerCrossCore, TestIntervalFaceting, DistribJoinFromCollectionTest, HdfsRestartWhileUpdatingTest, TestPointFields, TestConfigSets, TestDynamicLoading, DisMaxRequestHandlerTest, TestRawTransformer, TestShardHandlerFactory, RulesTest, UUIDFieldTest, TestLegacyFieldReuse, RequestLoggingTest, TestDownShardTolerantSearch, TestBlendedInfixSuggestions, ShardSplitTest, ActionThrottleTest, HLLUtilTest, CdcrWithNodesRestartsTest, DistributedMLTComponentTest, RankQueryTest, TestRTGBase, ClusterStateTest, DirectSolrConnectionTest, DistributedTermsComponentTest, RecoveryAfterSoftCommitTest, TestCursorMarkWithoutUniqueKey, PolyFieldTest, TestReplicaProperties, PluginInfoTest, TestManagedSynonymGraphFilterFactory, TestChildDocTransformer, TestStressCloudBlindAtomicUpdates, TestComponentsName, TestSQLHandler, SchemaVersionSpecificBehaviorTest, DocValuesNotIndexedTest, HdfsNNFailoverTest, SaslZkACLProviderTest, PeerSyncWithLeaderAndIndexFingerprintCachingTest, V2StandaloneTest, PeerSyncTest, TestSolrQueryParser, TestFieldSortValues, TestIndexingPerformance, TestPHPSerializedResponseWriter, SuggesterWFSTTest, TestStressLucene, FastVectorHighlighterTest, PhrasesIdentificationComponentTest, TestDFISimilarityFactory, PrimitiveFieldTypeTest, CoreMergeIndexesAdminHandlerTest, TestMaxTokenLenTokenizer, TestLockTree, VersionInfoTest, TestAuthorizationFramework, AssignBackwardCompatibilityTest, BasicDistributedZkTest, BasicZkTest, ChaosMonkeyNothingIsSafeTest, ChaosMonkeySafeLeaderTest, ConnectionManagerTest, DeleteInactiveReplicaTest, DistributedVersionInfoTest, LeaderElectionTest, LeaderFailoverAfterPartitionTest, LeaderFailureAfterFreshStartTest, OverseerModifyCollectionTest, OverseerTaskQueueTest, PeerSyncReplicationTest, TestCloudConsistency, TestConfigSetsAPIExclusivity, TestConfigSetsAPIZkFailure, TestCryptoKeys, TestDistributedMap, TestHashPartitioner, TestLeaderElectionZkExpiry]
[junit4] Completed [513/838 (1!)] on J1 in 71.05s, 1 test, 1 failure <<< FAILURES!
[...truncated 43636 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3509/consoleText
[repro] Revision: 2835e086537ee708dc6037e19b33c5aa3c287a15
[repro] Ant options: "-Dargs=-XX:-UseCompressedOops -XX:+UseG1GC"
[repro] Repro line: ant test -Dtestcase=TestLeaderElectionZkExpiry -Dtests.method=testLeaderElectionWithZkExpiry -Dtests.seed=F49C4FF42EEF100B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=dsb -Dtests.timezone=America/Asuncion -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[repro] ant clean
[...truncated 6 lines...]
[repro] Test suites by module:
[repro] solr/core
[repro] TestLeaderElectionZkExpiry
[repro] ant compile-test
[...truncated 3588 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.TestLeaderElectionZkExpiry" -Dtests.showOutput=onerror "-Dargs=-XX:-UseCompressedOops -XX:+UseG1GC" -Dtests.seed=F49C4FF42EEF100B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=dsb -Dtests.timezone=America/Asuncion -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[...truncated 75 lines...]
[repro] Failures:
[repro] 0/5 failed: org.apache.solr.cloud.TestLeaderElectionZkExpiry
[repro] Exiting with code 0
[...truncated 43 lines...]