You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2015/12/19 19:22:48 UTC
[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 3852 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/3852/
1 tests failed.
FAILED: org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior
Error Message:
Illegal state, was: down expected:active clusterState:live nodes:[]collections:{c1=DocCollection(c1)={ "shards":{"shard1":{ "state":"active", "range":null, "parent":null, "replicas":{"core_node1":{ "base_url":"http://127.0.0.1/solr", "node_name":"node1", "core":"core1", "roles":"", "state":"down"}}}}, "router":{"name":"implicit"}}, test=LazyCollectionRef(test)}
Stack Trace:
java.lang.AssertionError: Illegal state, was: down expected:active clusterState:live nodes:[]collections:{c1=DocCollection(c1)={
"shards":{"shard1":{
"state":"active",
"range":null,
"parent":null,
"replicas":{"core_node1":{
"base_url":"http://127.0.0.1/solr",
"node_name":"node1",
"core":"core1",
"roles":"",
"state":"down"}}}},
"router":{"name":"implicit"}}, test=LazyCollectionRef(test)}
at __randomizedtesting.SeedInfo.seed([1928F325F304463C:7136F0C911941C72]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractDistribZkTestBase.verifyReplicaStatus(AbstractDistribZkTestBase.java:237)
at org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior(OverseerTest.java:1262)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 9810 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J2/temp/solr.cloud.OverseerTest_1928F325F304463C-001/init-core-data-001
[junit4] 2> 6909 INFO (SUITE-OverseerTest-seed#[1928F325F304463C]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
[junit4] 2> 6909 INFO (SUITE-OverseerTest-seed#[1928F325F304463C]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore
[junit4] 2> 6909 INFO (SUITE-OverseerTest-seed#[1928F325F304463C]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore end
[junit4] 2> 6937 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testShardLeaderChange
[junit4] 2> 6956 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 6964 INFO (Thread-4) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 6964 INFO (Thread-4) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 7167 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.ZkTestServer start zk server on port:60416
[junit4] 2> 7219 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 7327 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 7373 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@19fa7c4 name:ZooKeeperConnection Watcher:127.0.0.1:60416/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 7373 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 7374 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 7376 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 7382 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 7393 INFO (zkCallback-3-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@65ce0806 name:ZooKeeperConnection Watcher:127.0.0.1:60416 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 7393 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 7394 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 7419 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 7426 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 7427 INFO (zkCallback-4-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1e79c125 name:ZooKeeperConnection Watcher:127.0.0.1:60416 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 7427 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 7427 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 7429 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 7458 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 7461 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 7465 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 7468 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 7472 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 7476 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 7482 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 7483 INFO (Thread-5) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 7485 INFO (zkCallback-5-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@549154f9 name:ZooKeeperConnection Watcher:127.0.0.1:60416/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 7485 INFO (Thread-5) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 7485 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 7494 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 7495 INFO (Thread-5) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 7503 INFO (zkCallback-6-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@578e5065 name:ZooKeeperConnection Watcher:127.0.0.1:60416/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 7504 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 7504 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 7514 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 7516 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
[junit4] 2> 7521 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
[junit4] 2> 7526 INFO (Thread-5) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 7528 INFO (Thread-5) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95063054811332611-127.0.0.1:60416_solr-n_0000000000
[junit4] 2> 7537 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 7541 INFO (Thread-5) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60416_solr
[junit4] 2> 7547 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 7551 INFO (zkCallback-6-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 7556 INFO (Thread-5) [ ] o.a.s.c.Overseer Overseer (id=95063054811332611-127.0.0.1:60416_solr-n_0000000000) starting
[junit4] 2> 7563 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
[junit4] 2> 7571 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
[junit4] 2> 7577 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
[junit4] 2> 7582 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
[junit4] 2> 7593 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
[junit4] 2> 7617 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
[junit4] 2> 7647 INFO (Thread-5) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 7755 INFO (OverseerCollectionConfigSetProcessor-95063054811332611-127.0.0.1:60416_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 7757 INFO (Thread-5) [ ] o.a.s.c.OverseerTest Killing overseer.
[junit4] 2> 7755 INFO (OverseerStateUpdate-95063054811332611-127.0.0.1:60416_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 7762 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 7768 INFO (Thread-5) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 7770 INFO (zkCallback-8-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@139bc376 name:ZooKeeperConnection Watcher:127.0.0.1:60416/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 7770 INFO (Thread-5) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 7770 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 7771 INFO (Thread-5) [ ] o.a.s.c.Overseer Overseer (id=95063054811332611-127.0.0.1:60416_solr-n_0000000000) closing
[junit4] 2> 7771 INFO (Thread-5) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 7771 INFO (OverseerStateUpdate-95063054811332611-127.0.0.1:60416_solr-n_0000000000) [ ] o.a.s.c.Overseer According to ZK I (id=95063054811332611-127.0.0.1:60416_solr-n_0000000000) am no longer a leader.
[junit4] 2> 7771 INFO (OverseerStateUpdate-95063054811332611-127.0.0.1:60416_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:60416_solr
[junit4] 2> 7774 ERROR (OverseerCollectionConfigSetProcessor-95063054811332611-127.0.0.1:60416_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Unable to prioritize overseer
[junit4] 2> java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.cloud.OverseerNodePrioritizer.prioritizeOverseerNodes(OverseerNodePrioritizer.java:61)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:159)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 7783 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:305)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:88)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:266)
[junit4] 2> 7783 ERROR (OverseerHdfsCoreFailoverThread-95063054811332611-127.0.0.1:60416_solr-n_0000000000) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread OverseerAutoReplicaFailoverThread had an error in its thread work loop.:org.apache.solr.common.SolrException: Error reading cluster properties
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getClusterProps(ZkStateReader.java:738)
[junit4] 2> at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.doWork(OverseerAutoReplicaFailoverThread.java:153)
[junit4] 2> at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:132)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getClusterProps(ZkStateReader.java:731)
[junit4] 2> ... 3 more
[junit4] 2>
[junit4] 2> 7787 INFO (Thread-5) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 7788 INFO (Thread-5) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95063054811332613-127.0.0.1:60416_solr-n_0000000001
[junit4] 2> 7789 INFO (Thread-5) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60416_solr
[junit4] 2> 7789 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 7792 INFO (Thread-5) [ ] o.a.s.c.Overseer Overseer (id=95063054811332613-127.0.0.1:60416_solr-n_0000000001) starting
[junit4] 2> 7810 INFO (Thread-5) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 7811 INFO (OverseerCollectionConfigSetProcessor-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 7811 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 7813 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 7826 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 0
[junit4] 2> 7836 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 7837 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
[junit4] 2> 7848 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 7856 INFO (zkCallback-6-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 7856 INFO (zkCallback-8-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 8070 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 8073 INFO (zkCallback-8-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 8073 INFO (zkCallback-6-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 8084 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95063054811332612-node1_core1-n_0000000000
[junit4] 2> 8091 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard1
[junit4] 2> 8107 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 8134 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 8140 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 8143 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 1
[junit4] 2> 8144 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 8144 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 8152 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95063054811332612-node1_core1-n_0000000001
[junit4] 2> 8153 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/95063054811332612-node1_core1-n_0000000001 , seq(0) /collections/collection1/leader_elect/shard1/election/95063054811332612-node1_core1-n_0000000000
[junit4] 2> 8154 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 8154 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 8157 INFO (zkCallback-10-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ad28f39 name:ZooKeeperConnection Watcher:127.0.0.1:60416/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 8157 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 8157 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 8161 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 8171 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node2
[junit4] 2> 8174 INFO (zkCallback-8-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 8174 INFO (zkCallback-6-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 8175 INFO (zkCallback-10-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 8181 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 8184 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 1
[junit4] 2> 8185 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 8185 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 8189 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95063054811332612-node1_core1-n_0000000002
[junit4] 2> 8191 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/95063054811332612-node1_core1-n_0000000002 , seq(0) /collections/collection1/leader_elect/shard1/election/95063054811332612-node1_core1-n_0000000000
[junit4] 2> 8191 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 8294 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 8294 INFO (zkCallback-8-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 8294 INFO (zkCallback-6-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 8294 INFO (zkCallback-10-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 8404 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"} current state version: 2
[junit4] 2> 8405 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"}
[junit4] 2> 8405 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=1
[junit4] 2> 8405 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 8511 INFO (zkCallback-8-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 8511 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 8511 INFO (zkCallback-6-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 8512 INFO (zkCallback-10-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 8902 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 8910 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95063054811332614-node2_core4-n_0000000003
[junit4] 2> 8917 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/95063054811332612-node1_core1-n_0000000002 to know if I could be the leader
[junit4] 2> 8918 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 8918 INFO (zkCallback-8-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 8919 INFO (zkCallback-6-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 8922 INFO (zkCallback-10-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 8926 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x151bb583d4b0004, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 8930 INFO (zkCallback-10-thread-2) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 8941 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node2/solr/",
[junit4] 2> "core":"core4",
[junit4] 2> "state":"active"} current state version: 3
[junit4] 2> 9048 INFO (zkCallback-10-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 9048 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 9048 INFO (zkCallback-8-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 9132 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 9132 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 9134 INFO (zkCallback-11-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5173b4cf name:ZooKeeperConnection Watcher:127.0.0.1:60416/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 9135 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 9135 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 9139 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 9146 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 9151 INFO (zkCallback-10-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 9151 INFO (zkCallback-8-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 9152 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 9152 INFO (zkCallback-11-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 9157 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 4
[junit4] 2> 9158 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 9158 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 9161 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 9170 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95063054811332615-node1_core1-n_0000000004
[junit4] 2> 9172 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/95063054811332614-node2_core4-n_0000000003 to know if I could be the leader
[junit4] 2> 9173 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 9173 INFO (zkCallback-8-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 9174 INFO (zkCallback-10-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 9182 INFO (zkCallback-11-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 9184 INFO (zkCallback-11-thread-2) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 9190 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1",
[junit4] 2> "state":"active"} current state version: 4
[junit4] 2> 9219 INFO (Thread-5) [ ] o.a.s.c.OverseerTest Killing overseer.
[junit4] 2> 9220 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 9221 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x151bb583d4b0005, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 9222 INFO (Thread-5) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 9224 INFO (zkCallback-12-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@79ca3b13 name:ZooKeeperConnection Watcher:127.0.0.1:60416/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 9224 INFO (Thread-5) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 9224 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 9224 INFO (Thread-5) [ ] o.a.s.c.Overseer Overseer (id=95063054811332613-127.0.0.1:60416_solr-n_0000000001) closing
[junit4] 2> 9224 INFO (OverseerStateUpdate-95063054811332613-127.0.0.1:60416_solr-n_0000000001) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:60416_solr
[junit4] 2> 9225 INFO (Thread-5) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 9227 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:305)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:88)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:266)
[junit4] 2> 9230 INFO (Thread-5) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 9232 INFO (Thread-5) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95063054811332616-127.0.0.1:60416_solr-n_0000000002
[junit4] 2> 9232 INFO (Thread-5) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60416_solr
[junit4] 2> 9233 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 9235 INFO (Thread-5) [ ] o.a.s.c.Overseer Overseer (id=95063054811332616-127.0.0.1:60416_solr-n_0000000002) starting
[junit4] 2> 9253 INFO (Thread-5) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 9254 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 9254 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 9260 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: workQueueSize: 2, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 9261 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 9261 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 9263 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 9265 INFO (zkCallback-12-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 9265 INFO (OverseerCollectionConfigSetProcessor-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 9265 INFO (zkCallback-11-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 9271 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: workQueueSize: 0, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1",
[junit4] 2> "state":"active"}
[junit4] 2> 9272 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 9273 INFO (zkCallback-11-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 9273 INFO (zkCallback-12-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 10674 WARN (zkCallback-10-thread-2) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 10678 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 10680 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 6
[junit4] 2> 10680 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 10681 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 10685 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95063054811332615-node1_core1-n_0000000005
[junit4] 2> 10686 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/95063054811332615-node1_core1-n_0000000005 , seq(0) /collections/collection1/leader_elect/shard1/election/95063054811332615-node1_core1-n_0000000004
[junit4] 2> 10686 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 10690 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 10691 INFO (zkCallback-14-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7791e67 name:ZooKeeperConnection Watcher:127.0.0.1:60416/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 10691 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 10692 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 10695 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 10700 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node2
[junit4] 2> 10702 INFO (zkCallback-12-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 10702 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 10702 INFO (zkCallback-14-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 10702 INFO (zkCallback-11-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 10707 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 10709 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 6
[junit4] 2> 10709 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 10709 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 10714 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95063054811332615-node1_core1-n_0000000006
[junit4] 2> 10715 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/95063054811332615-node1_core1-n_0000000006 , seq(0) /collections/collection1/leader_elect/shard1/election/95063054811332615-node1_core1-n_0000000004
[junit4] 2> 10721 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 10722 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"} current state version: 6
[junit4] 2> 10723 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"}
[junit4] 2> 10723 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 10728 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95063054811332617-node2_core4-n_0000000007
[junit4] 2> 10733 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/95063054811332615-node1_core1-n_0000000006 to know if I could be the leader
[junit4] 2> 10735 INFO (zkCallback-11-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 10735 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 10735 INFO (zkCallback-14-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 10735 INFO (zkCallback-12-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 10740 INFO (zkCallback-14-thread-2) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 10746 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node2/solr/",
[junit4] 2> "core":"core4",
[junit4] 2> "state":"active"} current state version: 6
[junit4] 2> 10855 INFO (zkCallback-12-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 10855 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 10856 INFO (zkCallback-14-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 10943 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 10944 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 10946 INFO (zkCallback-15-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1bbe65a name:ZooKeeperConnection Watcher:127.0.0.1:60416/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 10946 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 10947 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 10952 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 10960 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 10962 INFO (zkCallback-14-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 10963 INFO (zkCallback-12-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 10963 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 10964 INFO (zkCallback-15-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 10968 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 10969 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 7
[junit4] 2> 10969 INFO (Thread-5) [ ] o.a.s.c.OverseerTest Killing overseer.
[junit4] 2> 10971 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 10971 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 10974 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 10975 INFO (Thread-5) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 10977 INFO (zkCallback-16-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5f3c9009 name:ZooKeeperConnection Watcher:127.0.0.1:60416/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 10978 INFO (Thread-5) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 10978 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 10979 INFO (Thread-5) [ ] o.a.s.c.Overseer Overseer (id=95063054811332616-127.0.0.1:60416_solr-n_0000000002) closing
[junit4] 2> 10979 INFO (OverseerStateUpdate-95063054811332616-127.0.0.1:60416_solr-n_0000000002) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:60416_solr
[junit4] 2> 10979 INFO (Thread-5) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 10983 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:305)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:88)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:266)
[junit4] 2> 10983 INFO (Thread-5) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 10985 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95063054811332618-node1_core1-n_0000000008
[junit4] 2> 10985 INFO (Thread-5) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95063054811332619-127.0.0.1:60416_solr-n_0000000003
[junit4] 2> 10987 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/95063054811332617-node2_core4-n_0000000007 to know if I could be the leader
[junit4] 2> 10988 INFO (zkCallback-14-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 10988 INFO (zkCallback-15-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 10989 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 10989 INFO (Thread-5) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60416_solr
[junit4] 2> 10990 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 10990 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x151bb583d4b0009, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 10994 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 10999 INFO (Thread-5) [ ] o.a.s.c.Overseer Overseer (id=95063054811332619-127.0.0.1:60416_solr-n_0000000003) starting
[junit4] 2> 11000 INFO (zkCallback-15-thread-2) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 11007 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95063054811332618-node1_core1-n_0000000009
[junit4] 2> 11008 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/95063054811332618-node1_core1-n_0000000009 , seq(0) /collections/collection1/leader_elect/shard1/election/95063054811332618-node1_core1-n_0000000008
[junit4] 2> 11008 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 11014 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 11018 INFO (zkCallback-18-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@dd3c0f3 name:ZooKeeperConnection Watcher:127.0.0.1:60416/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 11018 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 11019 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 11020 INFO (Thread-5) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 11021 INFO (OverseerCollectionConfigSetProcessor-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 11027 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 11031 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 11032 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 11037 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node2
[junit4] 2> 11039 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 7
[junit4] 2> 11040 INFO (zkCallback-18-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 11040 INFO (zkCallback-16-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 11040 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 11041 INFO (zkCallback-15-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 11042 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 11042 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 11046 INFO (zkCallback-16-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11046 INFO (zkCallback-15-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11047 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11047 INFO (zkCallback-18-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11050 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 11063 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.Overseer processMessage: queueSize: 0, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 8
[junit4] 2> 11065 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 11065 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95063054811332618-node1_core1-n_0000000010
[junit4] 2> 11066 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/95063054811332618-node1_core1-n_0000000010 , seq(0) /collections/collection1/leader_elect/shard1/election/95063054811332618-node1_core1-n_0000000008
[junit4] 2> 11065 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 11071 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.Overseer processMessage: queueSize: 0, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1",
[junit4] 2> "state":"active"} current state version: 8
[junit4] 2> 11077 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 8
[junit4] 2> 11078 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 11078 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 11183 INFO (zkCallback-18-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11183 INFO (zkCallback-16-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11184 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11184 INFO (zkCallback-15-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11283 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 11286 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"} current state version: 9
[junit4] 2> 11287 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"}
[junit4] 2> 11287 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 11292 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95063054811332620-node2_core4-n_0000000011
[junit4] 2> 11294 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/95063054811332618-node1_core1-n_0000000010 to know if I could be the leader
[junit4] 2> 11295 INFO (zkCallback-16-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11295 INFO (zkCallback-18-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11295 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11295 INFO (zkCallback-15-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11300 INFO (zkCallback-18-thread-1) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 11308 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node2/solr/",
[junit4] 2> "core":"core4",
[junit4] 2> "state":"active"} current state version: 9
[junit4] 2> 11414 INFO (zkCallback-18-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 11414 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 11415 INFO (zkCallback-16-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 12795 WARN (zkCallback-15-thread-2) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 12798 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 12799 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 12801 INFO (zkCallback-19-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5fd70f7e name:ZooKeeperConnection Watcher:127.0.0.1:60416/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 12801 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 12802 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 12805 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 12810 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 12812 INFO (zkCallback-18-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 12812 INFO (zkCallback-2-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 12812 INFO (zkCallback-16-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 12813 INFO (zkCallback-19-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 12817 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 12818 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 10
[junit4] 2> 12819 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 12819 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 12826 INFO (Thread-5) [ ] o.a.s.c.OverseerTest Killing overseer.
[junit4] 2> 12828 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x151bb583d4b000b, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 12828 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 12829 INFO (Thread-5) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 12829 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/95063054811332621-node1_core1-n_0000000012
[junit4] 2> 12831 INFO (zkCallback-20-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@dfa0111 name:ZooKeeperConnection Watcher:127.0.0.1:60416/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 12831 INFO (Thread-5) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 12832 INFO (Thread-5) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 12832 INFO (Thread-5) [ ] o.a.s.c.Overseer Overseer (id=95063054811332619-127.0.0.1:60416_solr-n_0000000003) closing
[junit4] 2> 12832 INFO (OverseerStateUpdate-95063054811332619-127.0.0.1:60416_solr-n_0000000003) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:60416_solr
[junit4] 2> 12832 INFO (Thread-5) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 12833 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/95063054811332620-node2_core4-n_0000000011 to know if I could be t
[...truncated too long message...]
collection":"collection2",
[junit4] 2> "base_url":"http://127.0.0.1/solr/",
[junit4] 2> "core":"core2",
[junit4] 2> "state":"active"} current state version: 6
[junit4] 2> 65397 INFO (OverseerStateUpdate-95063058433441795-127.0.0.1:51416_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"} current state version: 6
[junit4] 2> 65398 INFO (OverseerStateUpdate-95063058433441795-127.0.0.1:51416_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 65398 INFO (OverseerStateUpdate-95063058433441795-127.0.0.1:51416_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=3
[junit4] 2> 65398 INFO (OverseerStateUpdate-95063058433441795-127.0.0.1:51416_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 65501 INFO (zkCallback-112-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 65501 INFO (zkCallback-113-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 65501 INFO (zkCallback-115-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 65897 INFO (TEST-OverseerTest.testBadQueueItem-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection2/leader_elect/shard1/election
[junit4] 2> 65901 INFO (TEST-OverseerTest.testBadQueueItem-seed#[1928F325F304463C]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection2/leader_elect/shard1/election/95063058433441796-127.0.0.1_core3-n_0000000000
[junit4] 2> 65902 INFO (TEST-OverseerTest.testBadQueueItem-seed#[1928F325F304463C]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection2/leaders/shard1
[junit4] 2> 65904 INFO (TEST-OverseerTest.testBadQueueItem-seed#[1928F325F304463C]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 65907 INFO (OverseerStateUpdate-95063058433441795-127.0.0.1:51416_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection2",
[junit4] 2> "base_url":"http://127.0.0.1/solr/",
[junit4] 2> "core":"core3",
[junit4] 2> "state":"active"} current state version: 7
[junit4] 2> 66010 INFO (zkCallback-112-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 66011 INFO (zkCallback-113-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 66010 INFO (zkCallback-115-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 66062 INFO (zkCallback-113-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 66062 INFO (zkCallback-115-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 66066 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x151bb5915300004, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 67562 WARN (zkCallback-115-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 67565 INFO (TEST-OverseerTest.testBadQueueItem-seed#[1928F325F304463C]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51416 51416
[junit4] 2> 67720 INFO (Thread-15) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51416 51416
[junit4] 2> 67721 WARN (Thread-15) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/collections
[junit4] 2> 3 /solr/live_nodes
[junit4] 2>
[junit4] 2> 67722 INFO (TEST-OverseerTest.testBadQueueItem-seed#[1928F325F304463C]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testBadQueueItem
[junit4] 2> 67722 INFO (TEST-OverseerTest.testBadQueueItem-seed#[1928F325F304463C]) [ ] o.a.s.c.Overseer Overseer (id=95063058433441795-127.0.0.1:51416_solr-n_0000000000) closing
[junit4] 2> 67724 INFO (OverseerStateUpdate-95063058433441795-127.0.0.1:51416_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:51416_solr
[junit4] 2> 67746 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:305)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:88)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:266)
[junit4] 2> 70724 INFO (SUITE-OverseerTest-seed#[1928F325F304463C]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J2/temp/solr.cloud.OverseerTest_1928F325F304463C-001
[junit4] 2> NOTE: test params are: codec=Lucene54, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=ar_BH, timezone=Pacific/Efate
[junit4] 2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_80 (64-bit)/cpus=4,threads=1,free=307868432,total=338165760
[junit4] 2> NOTE: All tests run in this JVM: [TestComplexPhraseQParserPlugin, OverseerTest]
[junit4] Completed [6/555 (1!)] on J2 in 63.98s, 12 tests, 1 failure, 1 skipped <<< FAILURES!
[...truncated 1735 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:794: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:738: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/common-build.xml:526: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/common-build.xml:1456: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/common-build.xml:1010: There were test failures: 555 suites (7 ignored), 2206 tests, 1 failure, 53 ignored (38 assumptions) [seed: 1928F325F304463C]
Total time: 61 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any