You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/12/16 02:52:51 UTC
[JENKINS] Lucene-Solr-5.x-Solaris (64bit/jdk1.8.0) - Build # 260 -
Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Solaris/260/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
1 tests failed.
FAILED: org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior
Error Message:
Illegal state, was: down expected:active clusterState:live nodes:[]collections:{c1=DocCollection(c1)={ "shards":{"shard1":{ "parent":null, "range":null, "state":"active", "replicas":{"core_node1":{ "base_url":"http://127.0.0.1/solr", "node_name":"node1", "core":"core1", "roles":"", "state":"down"}}}}, "router":{"name":"implicit"}}, test=LazyCollectionRef(test)}
Stack Trace:
java.lang.AssertionError: Illegal state, was: down expected:active clusterState:live nodes:[]collections:{c1=DocCollection(c1)={
"shards":{"shard1":{
"parent":null,
"range":null,
"state":"active",
"replicas":{"core_node1":{
"base_url":"http://127.0.0.1/solr",
"node_name":"node1",
"core":"core1",
"roles":"",
"state":"down"}}}},
"router":{"name":"implicit"}}, test=LazyCollectionRef(test)}
at __randomizedtesting.SeedInfo.seed([F137852DF4029A5B:992986C11692C015]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractDistribZkTestBase.verifyReplicaStatus(AbstractDistribZkTestBase.java:237)
at org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior(OverseerTest.java:1262)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 10416 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerTest_F137852DF4029A5B-001/init-core-data-001
[junit4] 2> 1792924 INFO (SUITE-OverseerTest-seed#[F137852DF4029A5B]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
[junit4] 2> 1792924 INFO (SUITE-OverseerTest-seed#[F137852DF4029A5B]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore
[junit4] 2> 1792924 INFO (SUITE-OverseerTest-seed#[F137852DF4029A5B]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore end
[junit4] 2> 1792927 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testShardLeaderChange
[junit4] 2> 1792928 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1792928 INFO (Thread-5809) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1792928 INFO (Thread-5809) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1793028 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.ZkTestServer start zk server on port:35652
[junit4] 2> 1793028 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1793029 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1793034 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7c14b12f name:ZooKeeperConnection Watcher:127.0.0.1:35652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1793034 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1793034 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1793034 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1793035 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1793043 INFO (zkCallback-1150-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@46bc2fff name:ZooKeeperConnection Watcher:127.0.0.1:35652 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1793043 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1793044 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1793045 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1793046 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1793051 INFO (zkCallback-1151-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3af0e37 name:ZooKeeperConnection Watcher:127.0.0.1:35652 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1793051 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1793051 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1793052 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 1793056 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 1793058 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 1793060 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 1793062 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 1793064 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 1793070 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1793071 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1793071 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1793073 INFO (Thread-5810) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1793080 INFO (zkCallback-1152-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@60966fdb name:ZooKeeperConnection Watcher:127.0.0.1:35652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1793080 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1793080 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1793084 INFO (zkCallback-1153-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4ccc4ca3 name:ZooKeeperConnection Watcher:127.0.0.1:35652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1793084 INFO (Thread-5810) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1793084 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1793084 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1793084 INFO (Thread-5810) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 1793088 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
[junit4] 2> 1793088 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 1793090 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
[junit4] 2> 1793090 INFO (zkCallback-1152-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1793092 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
[junit4] 2> 1793093 INFO (Thread-5810) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1793094 INFO (Thread-5810) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95042222919385092-127.0.0.1:35652_solr-n_0000000000
[junit4] 2> 1793094 INFO (Thread-5810) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35652_solr
[junit4] 2> 1793094 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 1793096 INFO (Thread-5810) [ ] o.a.s.c.Overseer Overseer (id=95042222919385092-127.0.0.1:35652_solr-n_0000000000) starting
[junit4] 2> 1793100 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
[junit4] 2> 1793103 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
[junit4] 2> 1793106 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
[junit4] 2> 1793108 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
[junit4] 2> 1793111 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
[junit4] 2> 1793114 INFO (Thread-5810) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1793115 INFO (Thread-5810) [ ] o.a.s.c.OverseerTest Killing overseer.
[junit4] 2> 1793116 INFO (OverseerCollectionConfigSetProcessor-95042222919385092-127.0.0.1:35652_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 1793118 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1793119 INFO (Thread-5810) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1793119 ERROR (OverseerHdfsCoreFailoverThread-95042222919385092-127.0.0.1:35652_solr-n_0000000000) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread OverseerAutoReplicaFailoverThread had an error in its thread work loop.:org.apache.solr.common.SolrException: Error reading cluster properties
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getClusterProps(ZkStateReader.java:738)
[junit4] 2> at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.doWork(OverseerAutoReplicaFailoverThread.java:153)
[junit4] 2> at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:132)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterprops.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getClusterProps(ZkStateReader.java:731)
[junit4] 2> ... 3 more
[junit4] 2>
[junit4] 2> 1793127 INFO (zkCallback-1155-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4f7cadcd name:ZooKeeperConnection Watcher:127.0.0.1:35652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1793127 INFO (Thread-5810) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1793127 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1793127 INFO (Thread-5810) [ ] o.a.s.c.Overseer Overseer (id=95042222919385092-127.0.0.1:35652_solr-n_0000000000) closing
[junit4] 2> 1793128 INFO (Thread-5810) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 1793128 INFO (OverseerCollectionConfigSetProcessor-95042222919385092-127.0.0.1:35652_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=95042222919385092-127.0.0.1:35652_solr-n_0000000000) am no longer a leader.
[junit4] 2> 1793128 INFO (OverseerStateUpdate-95042222919385092-127.0.0.1:35652_solr-n_0000000000) [ ] o.a.s.c.Overseer According to ZK I (id=95042222919385092-127.0.0.1:35652_solr-n_0000000000) am no longer a leader.
[junit4] 2> 1793128 INFO (OverseerStateUpdate-95042222919385092-127.0.0.1:35652_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 1793128 INFO (OverseerStateUpdate-95042222919385092-127.0.0.1:35652_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:35652_solr
[junit4] 2> 1793128 ERROR (OverseerCollectionConfigSetProcessor-95042222919385092-127.0.0.1:35652_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/collection-queue-work
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:325)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:325)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:312)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskQueue.getTailId(OverseerTaskQueue.java:272)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:144)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1793129 ERROR (OverseerCollectionConfigSetProcessor-95042222919385092-127.0.0.1:35652_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Unable to prioritize overseer
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.cloud.OverseerNodePrioritizer.prioritizeOverseerNodes(OverseerNodePrioritizer.java:61)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:159)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1793133 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:305)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:88)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:266)
[junit4] 2> 1793135 INFO (Thread-5810) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1793136 INFO (Thread-5810) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95042222919385093-127.0.0.1:35652_solr-n_0000000001
[junit4] 2> 1793136 INFO (Thread-5810) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35652_solr
[junit4] 2> 1793136 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 1793138 INFO (Thread-5810) [ ] o.a.s.c.Overseer Overseer (id=95042222919385093-127.0.0.1:35652_solr-n_0000000001) starting
[junit4] 2> 1793148 INFO (Thread-5810) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1793149 INFO (OverseerCollectionConfigSetProcessor-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 1793149 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 1793150 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1793155 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 0
[junit4] 2> 1793156 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1793156 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
[junit4] 2> 1793156 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1793159 INFO (zkCallback-1152-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1793160 INFO (zkCallback-1155-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1793595 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1793597 INFO (zkCallback-1152-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 1793597 INFO (zkCallback-1155-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 1793603 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95042222919385091-node1_core1-n_0000000000
[junit4] 2> 1793605 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1793609 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 1793613 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 1793613 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1793616 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 1
[junit4] 2> 1793616 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1793616 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1793618 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95042222919385091-node1_core1-n_0000000001
[junit4] 2> 1793618 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/95042222919385091-node1_core1-n_0000000001 , seq(0) /collections/collection1/leader_elect/shard1/election/95042222919385091-node1_core1-n_0000000000
[junit4] 2> 1793619 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1793619 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1793621 INFO (zkCallback-1157-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@fb8ef8 name:ZooKeeperConnection Watcher:127.0.0.1:35652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1793621 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1793621 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1793623 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1793626 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node2
[junit4] 2> 1793628 INFO (zkCallback-1157-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1793628 INFO (zkCallback-1155-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1793628 INFO (zkCallback-1152-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1793630 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1793632 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 1
[junit4] 2> 1793632 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1793632 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1793636 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95042222919385091-node1_core1-n_0000000002
[junit4] 2> 1793636 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/95042222919385091-node1_core1-n_0000000002 , seq(0) /collections/collection1/leader_elect/shard1/election/95042222919385091-node1_core1-n_0000000000
[junit4] 2> 1793636 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1793736 INFO (zkCallback-1157-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1793736 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1793736 INFO (zkCallback-1155-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1793737 INFO (zkCallback-1152-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1793846 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"} current state version: 2
[junit4] 2> 1793846 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"}
[junit4] 2> 1793846 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=1
[junit4] 2> 1793846 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1793949 INFO (zkCallback-1155-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1793949 INFO (zkCallback-1157-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1793949 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1793950 INFO (zkCallback-1152-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1794344 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1794348 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95042222919385094-node2_core4-n_0000000003
[junit4] 2> 1794349 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/95042222919385091-node1_core1-n_0000000002 to know if I could be the leader
[junit4] 2> 1794350 INFO (zkCallback-1157-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1794350 INFO (zkCallback-1155-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1794350 INFO (zkCallback-1152-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1794351 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1794354 INFO (zkCallback-1157-thread-1) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 1794357 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node2/solr/",
[junit4] 2> "core":"core4",
[junit4] 2> "state":"active"} current state version: 3
[junit4] 2> 1794461 INFO (zkCallback-1157-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1794461 INFO (zkCallback-1155-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1794461 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1795853 WARN (zkCallback-1152-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1795856 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1795856 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1795858 INFO (zkCallback-1158-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@73cb4361 name:ZooKeeperConnection Watcher:127.0.0.1:35652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1795858 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1795858 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1795861 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1795863 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 1795865 INFO (zkCallback-1158-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1795865 INFO (zkCallback-1155-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1795865 INFO (zkCallback-1157-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1795865 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1795868 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1795869 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 4
[junit4] 2> 1795870 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1795871 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1795873 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95042222919385095-node1_core1-n_0000000004
[junit4] 2> 1795874 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/95042222919385094-node2_core4-n_0000000003 to know if I could be the leader
[junit4] 2> 1795874 INFO (zkCallback-1155-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1795874 INFO (zkCallback-1158-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1795874 INFO (zkCallback-1157-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1795875 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1795877 INFO (zkCallback-1158-thread-1) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 1795881 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1",
[junit4] 2> "state":"active"} current state version: 4
[junit4] 2> 1795956 INFO (Thread-5810) [ ] o.a.s.c.OverseerTest Killing overseer.
[junit4] 2> 1795957 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1795958 INFO (Thread-5810) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1795960 INFO (zkCallback-1159-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2cbf3ed3 name:ZooKeeperConnection Watcher:127.0.0.1:35652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1795960 INFO (Thread-5810) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1795960 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1795960 INFO (Thread-5810) [ ] o.a.s.c.Overseer Overseer (id=95042222919385093-127.0.0.1:35652_solr-n_0000000001) closing
[junit4] 2> 1795961 INFO (Thread-5810) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 1795962 INFO (OverseerStateUpdate-95042222919385093-127.0.0.1:35652_solr-n_0000000001) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:35652_solr
[junit4] 2> 1795963 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:305)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:88)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:266)
[junit4] 2> 1795966 INFO (Thread-5810) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1795966 INFO (Thread-5810) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95042222919385096-127.0.0.1:35652_solr-n_0000000002
[junit4] 2> 1795967 INFO (Thread-5810) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35652_solr
[junit4] 2> 1795967 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 1795969 INFO (Thread-5810) [ ] o.a.s.c.Overseer Overseer (id=95042222919385096-127.0.0.1:35652_solr-n_0000000002) starting
[junit4] 2> 1795979 INFO (Thread-5810) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1795980 INFO (OverseerCollectionConfigSetProcessor-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 1795980 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 1795981 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1795985 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: workQueueSize: 2, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1795985 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1795985 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1795986 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1795986 INFO (zkCallback-1158-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1795986 INFO (zkCallback-1159-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1795988 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: workQueueSize: 2, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1",
[junit4] 2> "state":"active"}
[junit4] 2> 1795989 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1795989 INFO (zkCallback-1159-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1795989 INFO (zkCallback-1158-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1797377 WARN (zkCallback-1157-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1797379 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1797380 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 6
[junit4] 2> 1797381 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1797381 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1797384 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95042222919385095-node1_core1-n_0000000005
[junit4] 2> 1797385 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/95042222919385095-node1_core1-n_0000000005 , seq(0) /collections/collection1/leader_elect/shard1/election/95042222919385095-node1_core1-n_0000000004
[junit4] 2> 1797385 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1797385 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1797387 INFO (zkCallback-1161-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6dbfe067 name:ZooKeeperConnection Watcher:127.0.0.1:35652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1797387 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1797387 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1797389 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1797392 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node2
[junit4] 2> 1797393 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1797393 INFO (zkCallback-1158-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1797393 INFO (zkCallback-1159-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1797393 INFO (zkCallback-1161-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1797396 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1797397 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 6
[junit4] 2> 1797398 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1797398 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1797401 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95042222919385095-node1_core1-n_0000000006
[junit4] 2> 1797401 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/95042222919385095-node1_core1-n_0000000006 , seq(0) /collections/collection1/leader_elect/shard1/election/95042222919385095-node1_core1-n_0000000004
[junit4] 2> 1797404 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1797406 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"} current state version: 6
[junit4] 2> 1797407 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"}
[junit4] 2> 1797407 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1797409 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95042222919385097-node2_core4-n_0000000007
[junit4] 2> 1797410 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/95042222919385095-node1_core1-n_0000000006 to know if I could be the leader
[junit4] 2> 1797411 INFO (zkCallback-1161-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1797411 INFO (zkCallback-1159-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1797411 INFO (zkCallback-1158-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1797411 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1797412 INFO (zkCallback-1161-thread-1) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 1797416 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node2/solr/",
[junit4] 2> "core":"core4",
[junit4] 2> "state":"active"} current state version: 6
[junit4] 2> 1797519 INFO (zkCallback-1159-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1797519 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1797519 INFO (zkCallback-1161-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1798912 WARN (zkCallback-1158-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1798914 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1798914 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1798916 INFO (zkCallback-1162-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2c65dfd name:ZooKeeperConnection Watcher:127.0.0.1:35652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1798916 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1798917 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1798919 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1798922 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 1798924 INFO (zkCallback-1159-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1798924 INFO (zkCallback-1161-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1798924 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1798924 INFO (zkCallback-1162-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1798926 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1798927 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 7
[junit4] 2> 1798928 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1798928 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1798931 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95042222919385098-node1_core1-n_0000000008
[junit4] 2> 1798932 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/95042222919385097-node2_core4-n_0000000007 to know if I could be the leader
[junit4] 2> 1798933 INFO (zkCallback-1159-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1798934 INFO (zkCallback-1162-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1798934 INFO (zkCallback-1161-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1798934 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1798936 INFO (zkCallback-1162-thread-2) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 1798939 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1",
[junit4] 2> "state":"active"} current state version: 7
[junit4] 2> 1798989 INFO (Thread-5810) [ ] o.a.s.c.OverseerTest Killing overseer.
[junit4] 2> 1798990 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1798990 INFO (Thread-5810) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1798992 INFO (zkCallback-1163-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@25223e0b name:ZooKeeperConnection Watcher:127.0.0.1:35652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1798992 INFO (Thread-5810) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1798992 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1798992 INFO (Thread-5810) [ ] o.a.s.c.Overseer Overseer (id=95042222919385096-127.0.0.1:35652_solr-n_0000000002) closing
[junit4] 2> 1798992 INFO (Thread-5810) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 1798992 INFO (OverseerStateUpdate-95042222919385096-127.0.0.1:35652_solr-n_0000000002) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:35652_solr
[junit4] 2> 1798997 INFO (Thread-5810) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1798997 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:305)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:88)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:266)
[junit4] 2> 1798998 INFO (Thread-5810) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95042222919385099-127.0.0.1:35652_solr-n_0000000003
[junit4] 2> 1798999 INFO (Thread-5810) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35652_solr
[junit4] 2> 1798999 INFO (Thread-5810) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 1799000 INFO (Thread-5810) [ ] o.a.s.c.Overseer Overseer (id=95042222919385099-127.0.0.1:35652_solr-n_0000000003) starting
[junit4] 2> 1799009 INFO (Thread-5810) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1799009 INFO (OverseerCollectionConfigSetProcessor-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 1799010 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 1799010 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1799014 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.Overseer processMessage: workQueueSize: 2, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1799015 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1799015 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1799016 INFO (zkCallback-1162-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1799016 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1799016 INFO (zkCallback-1163-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1799018 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.Overseer processMessage: workQueueSize: 2, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1",
[junit4] 2> "state":"active"}
[junit4] 2> 1799019 INFO (zkCallback-1163-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1799019 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1799020 INFO (zkCallback-1162-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1800438 WARN (zkCallback-1161-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1800440 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1800441 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 9
[junit4] 2> 1800442 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1800442 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1800446 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95042222919385098-node1_core1-n_0000000009
[junit4] 2> 1800446 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/95042222919385098-node1_core1-n_0000000009 , seq(0) /collections/collection1/leader_elect/shard1/election/95042222919385098-node1_core1-n_0000000008
[junit4] 2> 1800447 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1800447 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1800449 INFO (zkCallback-1165-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1bc0fefc name:ZooKeeperConnection Watcher:127.0.0.1:35652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1800449 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1800450 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1800452 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1800455 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node2
[junit4] 2> 1800457 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1800457 INFO (zkCallback-1162-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1800458 INFO (zkCallback-1163-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1800458 INFO (zkCallback-1165-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1800460 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1800461 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 9
[junit4] 2> 1800462 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1800462 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1800465 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95042222919385098-node1_core1-n_0000000010
[junit4] 2> 1800466 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/95042222919385098-node1_core1-n_0000000010 , seq(0) /collections/collection1/leader_elect/shard1/election/95042222919385098-node1_core1-n_0000000008
[junit4] 2> 1800469 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1800470 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"} current state version: 9
[junit4] 2> 1800471 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"}
[junit4] 2> 1800471 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1800474 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95042222919385100-node2_core4-n_0000000011
[junit4] 2> 1800475 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/95042222919385098-node1_core1-n_0000000010 to know if I could be the leader
[junit4] 2> 1800476 INFO (zkCallback-1163-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1800476 INFO (zkCallback-1162-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1800476 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1800476 INFO (zkCallback-1165-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1800482 INFO (zkCallback-1165-thread-2) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 1800486 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node2/solr/",
[junit4] 2> "core":"core4",
[junit4] 2> "state":"active"} current state version: 9
[junit4] 2> 1800594 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1800594 INFO (zkCallback-1165-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1800594 INFO (zkCallback-1163-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1801978 WARN (zkCallback-1162-thread-2) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1801981 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1801981 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1801983 INFO (zkCallback-1166-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@529f657 name:ZooKeeperConnection Watcher:127.0.0.1:35652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1801983 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1801984 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1801986 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1801989 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 1801990 INFO (zkCallback-1165-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1801991 INFO (zkCallback-1163-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1801991 INFO (zkCallback-1149-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1801991 INFO (zkCallback-1166-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1801993 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[F137852DF4029A5B]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1801995 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 10
[junit4] 2> 1801995 INFO (OverseerStateUpdate-95042222919385099-127.0.0.1:35652_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":
[...truncated too long message...]
] 2> 1863626 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[F137852DF4029A5B]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:39127 39127
[junit4] 2> 1871456 ERROR (OverseerHdfsCoreFailoverThread-95042227398180868-127.0.0.1:39127_solr-n_0000000000) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread OverseerAutoReplicaFailoverThread had an error in its thread work loop.:org.apache.solr.common.SolrException: Error reading cluster properties
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getClusterProps(ZkStateReader.java:738)
[junit4] 2> at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.doWork(OverseerAutoReplicaFailoverThread.java:153)
[junit4] 2> at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:132)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterprops.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getClusterProps(ZkStateReader.java:731)
[junit4] 2> ... 3 more
[junit4] 2>
[junit4] 2> 1881460 ERROR (OverseerHdfsCoreFailoverThread-95042227398180868-127.0.0.1:39127_solr-n_0000000000) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread OverseerAutoReplicaFailoverThread had an error in its thread work loop.:org.apache.solr.common.SolrException: Error reading cluster properties
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getClusterProps(ZkStateReader.java:738)
[junit4] 2> at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.doWork(OverseerAutoReplicaFailoverThread.java:153)
[junit4] 2> at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:132)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterprops.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getClusterProps(ZkStateReader.java:731)
[junit4] 2> ... 3 more
[junit4] 2>
[junit4] 2> 1883650 INFO (Thread-5820) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:39127 39127
[junit4] 2> 1883652 WARN (Thread-5820) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/collections
[junit4] 2> 3 /solr/live_nodes
[junit4] 2>
[junit4] 2> 1883652 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[F137852DF4029A5B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testDoubleAssignment
[junit4] 2> 1883653 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[F137852DF4029A5B]) [ ] o.a.s.c.Overseer Overseer (id=95042227398180868-127.0.0.1:39127_solr-n_0000000000) closing
[junit4] 2> 1886655 INFO (SUITE-OverseerTest-seed#[F137852DF4029A5B]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerTest_F137852DF4029A5B-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene54): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=fr, timezone=Pacific/Pago_Pago
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=3,threads=1,free=255802568,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [CloudMLTQParserTest, TestSolrConfigHandlerConcurrent, CleanupOldIndexTest, TestDynamicFieldResource, DeleteInactiveReplicaTest, OpenCloseCoreStressTest, CSVRequestHandlerTest, DocValuesMissingTest, CoreAdminRequestStatusTest, TestLMJelinekMercerSimilarityFactory, TestReversedWildcardFilterFactory, CloneFieldUpdateProcessorFactoryTest, TestBadConfig, TestReloadDeadlock, SolrInfoMBeanTest, TestImplicitCoreProperties, TestReloadAndDeleteDocs, TestSolrCloudWithKerberosAlt, FieldAnalysisRequestHandlerTest, TestReRankQParserPlugin, PluginInfoTest, DistributedTermsComponentTest, ConcurrentDeleteAndCreateCollectionTest, AssignTest, ResponseLogComponentTest, SolrPluginUtilsTest, CollectionTooManyReplicasTest, TestClassNameShortening, TestAddFieldRealTimeGet, TestCloudManagedSchema, TestScoreJoinQPScore, TestDistributedMissingSort, MinimalSchemaTest, TestReload, SpellingQueryConverterTest, TestPerFieldSimilarity, TestJsonFacets, TestCollectionAPI, DistributedIntervalFacetingTest, TestMaxScoreQueryParser, SpellCheckComponentTest, TestBM25SimilarityFactory, TestHdfsUpdateLog, HighlighterMaxOffsetTest, ZkStateWriterTest, TestFastWriter, TestFiltering, PKIAuthenticationIntegrationTest, AsyncMigrateRouteKeyTest, TestCollationField, TestSolrQueryParser, SharedFSAutoReplicaFailoverTest, TestRestoreCore, DistributedQueryComponentCustomSortTest, TestFastOutputStream, TestSolrQueryParserDefaultOperatorResource, CopyFieldTest, FullSolrCloudDistribCmdsTest, DistributedVersionInfoTest, ShowFileRequestHandlerTest, TestRangeQuery, TestCloudInspectUtil, OpenExchangeRatesOrgProviderTest, ZkNodePropsTest, DistributedExpandComponentTest, TestRawResponseWriter, TestQuerySenderListener, TestChildDocTransformer, DateMathParserTest, BlockCacheTest, ExitableDirectoryReaderTest, TestHashQParserPlugin, TestFieldTypeResource, RequestHandlersTest, TestRandomCollapseQParserPlugin, TestConfigSets, QueryResultKeyTest, DistributedSuggestComponentTest, CollectionsAPIDistributedZkTest, TestStressRecovery, HdfsThreadLeakTest, TestSimpleTrackingShardHandler, TestDefaultSearchFieldResource, BJQParserTest, NotRequiredUniqueKeyTest, LoggingHandlerTest, SearchHandlerTest, TestClassicSimilarityFactory, AnalyticsMergeStrategyTest, ExplicitHLLTest, AnalysisErrorHandlingTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestBinaryField, TestExpandComponent, TestSimpleQParserPlugin, SolrIndexSplitterTest, HttpPartitionTest, InfoHandlerTest, TestLRUStatsCache, TestManagedSchemaFieldResource, TestValueSourceCache, ReplicationFactorTest, RAMDirectoryFactoryTest, SpellCheckCollatorTest, SpatialHeatmapFacetsTest, TestRandomFaceting, TestRestManager, HLLSerializationTest, TestIndexSearcher, TestReplicaProperties, TestTrackingShardHandlerFactory, BasicFunctionalityTest, HdfsLockFactoryTest, TestDocBasedVersionConstraints, HdfsBasicDistributedZkTest, HdfsChaosMonkeySafeLeaderTest, TestTrie, TestSolrCLIRunExample, TestCrossCoreJoin, TermVectorComponentTest, TestPerFieldSimilarityWithDefaultOverride, FileUtilsTest, TestCoreDiscovery, TestRecovery, TestFoldingMultitermQuery, TestOrdValues, ZkSolrClientTest, TestUniqueKeyFieldResource, DefaultValueUpdateProcessorTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestSolrXml, TermVectorComponentDistributedTest, StatsComponentTest, TestConfigSetImmutable, SuggesterWFSTTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, OverseerTest]
[junit4] Completed [267/555 (1!)] on J1 in 93.74s, 12 tests, 1 failure, 1 skipped <<< FAILURES!
[...truncated 901 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:794: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:738: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build.xml:233: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/common-build.xml:526: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/lucene/common-build.xml:1456: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/lucene/common-build.xml:1010: There were test failures: 555 suites (7 ignored), 2201 tests, 1 failure, 53 ignored (38 assumptions) [seed: F137852DF4029A5B]
Total time: 79 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any