You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/11/07 00:32:09 UTC
[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_66) - Build #
5378 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/5378/
Java: 64bit/jdk1.8.0_66 -XX:+UseCompressedOops -XX:+UseSerialGC
1 tests failed.
FAILED: org.apache.solr.cloud.OverseerTest.testOverseerStatsReset
Error Message:
expected:<1> but was:<2>
Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
at __randomizedtesting.SeedInfo.seed([A8631044D0ECD97:A1D2D638D8D46E99]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.OverseerTest.testOverseerStatsReset(OverseerTest.java:722)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:866)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:902)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:875)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:777)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:822)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 9977 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.OverseerTest_A8631044D0ECD97-001\init-core-data-001
[junit4] 2> 860007 INFO (SUITE-OverseerTest-seed#[A8631044D0ECD97]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false)
[junit4] 2> 860007 INFO (SUITE-OverseerTest-seed#[A8631044D0ECD97]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore
[junit4] 2> 860009 INFO (SUITE-OverseerTest-seed#[A8631044D0ECD97]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore end
[junit4] 2> 860010 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testDoubleAssignment
[junit4] 2> 860011 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 860011 INFO (Thread-1483) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 860011 INFO (Thread-1483) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 860131 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer start zk server on port:58816
[junit4] 2> 860131 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 860133 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 860137 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5856f1cd name:ZooKeeperConnection Watcher:127.0.0.1:58816/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 860137 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 860137 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 860137 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 860141 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 860143 INFO (zkCallback-1356-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@71aff226 name:ZooKeeperConnection Watcher:127.0.0.1:58816 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 860143 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 860143 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 860147 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x150df0417c90001, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 860147 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 860148 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 860150 INFO (zkCallback-1357-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3ba89e94 name:ZooKeeperConnection Watcher:127.0.0.1:58816 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 860150 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 860150 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 860157 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 860165 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 860165 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 860167 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 860168 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 860170 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 860171 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 860173 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 860175 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 860176 INFO (zkCallback-1358-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@28a724dc name:ZooKeeperConnection Watcher:127.0.0.1:58816/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 860176 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 860176 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 860178 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 860180 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 860182 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 860182 INFO (zkCallback-1358-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 860183 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 860185 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 860186 INFO (zkCallback-1359-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4634f45f name:ZooKeeperConnection Watcher:127.0.0.1:58816/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 860187 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 860187 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 860187 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 860198 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
[junit4] 2> 860199 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
[junit4] 2> 860200 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 860200 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94820800846692356-127.0.0.1:58816_solr-n_0000000000
[junit4] 2> 860200 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58816_solr
[junit4] 2> 860200 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 860201 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820800846692356-127.0.0.1:58816_solr-n_0000000000) starting
[junit4] 2> 860201 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
[junit4] 2> 860204 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
[junit4] 2> 860205 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
[junit4] 2> 860206 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
[junit4] 2> 860208 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
[junit4] 2> 860209 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
[junit4] 2> 860214 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 860214 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 860215 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 860215 INFO (OverseerCollectionConfigSetProcessor-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 860219 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 0
[junit4] 2> 860219 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 860219 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
[junit4] 2> 860219 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 860220 INFO (zkCallback-1359-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 860220 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 860220 INFO (zkCallback-1358-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 860811 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 860812 INFO (zkCallback-1359-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 860812 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 860812 INFO (zkCallback-1358-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 860817 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94820800846692355-node1_core1-n_0000000000
[junit4] 2> 860818 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard1
[junit4] 2> 860820 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 860823 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "core":"core1"} current state version: 1
[junit4] 2> 860824 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "core":"core1"}
[junit4] 2> 860826 INFO (zkCallback-1359-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 860826 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 860826 INFO (zkCallback-1358-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 860827 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x150df0417c90003, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 860827 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1"} current state version: 1
[junit4] 2> 860949 INFO (zkCallback-1359-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 860949 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 862610 WARN (zkCallback-1358-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 862610 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 862611 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 862613 INFO (zkCallback-1361-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@577ee554 name:ZooKeeperConnection Watcher:127.0.0.1:58816/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 862613 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 862613 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 862617 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 862620 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 862622 INFO (zkCallback-1359-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 862622 INFO (zkCallback-1361-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 862622 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 862624 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 862625 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 2
[junit4] 2> 862625 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 862625 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 862627 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94820800846692357-node1_core1-n_0000000001
[junit4] 2> 862628 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 862632 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1"} current state version: 2
[junit4] 2> 862754 INFO (zkCallback-1359-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 862754 INFO (zkCallback-1361-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 862754 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 862757 INFO (zkCallback-1361-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 862759 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 864588 WARN (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer Solr cannot talk to ZK, exiting Overseer main queue loop
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue-work
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:336)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:333)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:333)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:310)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:289)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.removeFirst(DistributedQueue.java:378)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.poll(DistributedQueue.java:187)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:239)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 864588 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:58816_solr
[junit4] 2> 864589 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 864590 WARN (zkCallback-1361-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 864594 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58816 58816
[junit4] 2> 865066 INFO (OverseerCollectionConfigSetProcessor-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 865066 INFO (OverseerCollectionConfigSetProcessor-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=94820800846692356-127.0.0.1:58816_solr-n_0000000000) am no longer a leader.
[junit4] 2> 867539 INFO (Thread-1483) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58816 58816
[junit4] 2> 867541 WARN (Thread-1483) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/collections
[junit4] 2> 3 /solr/live_nodes
[junit4] 2>
[junit4] 2> 867541 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testDoubleAssignment
[junit4] 2> 867541 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820800846692356-127.0.0.1:58816_solr-n_0000000000) closing
[junit4] 2> 867545 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testOverseerFailure
[junit4] 2> 867550 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 867550 INFO (Thread-1484) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 867550 INFO (Thread-1484) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 867682 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer start zk server on port:58837
[junit4] 2> 867682 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 867683 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 867688 INFO (zkCallback-1362-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@460b500e name:ZooKeeperConnection Watcher:127.0.0.1:58837 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 867688 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 867688 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 867691 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 867693 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 867695 INFO (zkCallback-1363-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@700fb258 name:ZooKeeperConnection Watcher:127.0.0.1:58837 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 867695 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 867695 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 867695 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 867700 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 867701 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 867702 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@48835df9 name:ZooKeeperConnection Watcher:127.0.0.1:58837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 867702 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 867702 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 867704 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 867706 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 867708 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 867710 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 867711 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 867714 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 867716 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 867718 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 867719 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@62f08480 name:ZooKeeperConnection Watcher:127.0.0.1:58837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 867719 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 867719 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 867722 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 867725 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 867727 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 867727 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 867728 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 867731 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 867732 INFO (zkCallback-1366-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4fcf925 name:ZooKeeperConnection Watcher:127.0.0.1:58837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 867732 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 867732 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 867732 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 867736 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
[junit4] 2> 867738 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
[junit4] 2> 867741 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 867741 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94820801340833796-127.0.0.1:58837_solr-n_0000000000
[junit4] 2> 867742 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58837_solr
[junit4] 2> 867742 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 867743 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820801340833796-127.0.0.1:58837_solr-n_0000000000) starting
[junit4] 2> 867745 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
[junit4] 2> 867748 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
[junit4] 2> 867751 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
[junit4] 2> 867753 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
[junit4] 2> 867756 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
[junit4] 2> 867759 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
[junit4] 2> 867764 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 867765 INFO (OverseerCollectionConfigSetProcessor-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 867765 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 867766 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 869076 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 0
[junit4] 2> 869077 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 869077 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
[junit4] 2> 869077 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 869078 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 869078 INFO (zkCallback-1366-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 869078 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 869729 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 869730 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 869730 INFO (zkCallback-1366-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 869730 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 869736 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94820801340833795-node1_core1-n_0000000000
[junit4] 2> 869737 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard1
[junit4] 2> 869740 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 869744 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "core":"core1"} current state version: 1
[junit4] 2> 869744 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "core":"core1"}
[junit4] 2> 869745 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1"} current state version: 1
[junit4] 2> 869746 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 869746 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 1
[junit4] 2> 869748 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 869748 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 869750 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94820801340833795-node1_core1-n_0000000001
[junit4] 2> 869750 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/94820801340833795-node1_core1-n_0000000001 , seq(0) /collections/collection1/leader_elect/shard1/election/94820801340833795-node1_core1-n_0000000000
[junit4] 2> 869882 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 869882 INFO (zkCallback-1366-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 869882 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 870384 INFO (OverseerCollectionConfigSetProcessor-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 870384 INFO (OverseerCollectionConfigSetProcessor-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=94820801340833796-127.0.0.1:58837_solr-n_0000000000) am no longer a leader.
[junit4] 2> 871258 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 871263 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94820801340833795-node1_core1-n_0000000002
[junit4] 2> 871263 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/94820801340833795-node1_core1-n_0000000002 , seq(0) /collections/collection1/leader_elect/shard1/election/94820801340833795-node1_core1-n_0000000000
[junit4] 2> 871264 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 871265 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 871267 INFO (zkCallback-1368-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3087bc29 name:ZooKeeperConnection Watcher:127.0.0.1:58837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 871267 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 871268 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 871268 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820801340833796-127.0.0.1:58837_solr-n_0000000000) closing
[junit4] 2> 871268 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:58837_solr
[junit4] 2> 871268 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 871269 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 871274 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 871276 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94820801340833797-127.0.0.1:58837_solr-n_0000000001
[junit4] 2> 871276 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58837_solr
[junit4] 2> 871276 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 871278 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820801340833797-127.0.0.1:58837_solr-n_0000000001) starting
[junit4] 2> 871288 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 871288 INFO (OverseerCollectionConfigSetProcessor-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 871288 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 871290 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 871293 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 2
[junit4] 2> 871293 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 871293 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 871295 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 871295 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 871295 INFO (zkCallback-1368-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 871365 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"deletecore",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1"} current state version: 3
[junit4] 2> 871502 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.o.ZkStateWriter going to delete_collection /collections/collection1/state.json
[junit4] 2> 871509 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 871509 INFO (zkCallback-1368-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 871509 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 871509 INFO (zkCallback-1368-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 871509 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 871509 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 872163 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 872163 INFO (zkCallback-1368-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 872163 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 874126 WARN (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 874128 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58837 58837
[junit4] 2> 876521 INFO (OverseerCollectionConfigSetProcessor-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.OverseerTaskProcessor
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 876521 INFO (OverseerCollectionConfigSetProcessor-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=94820801340833797-127.0.0.1:58837_solr-n_0000000001) am no longer a leader.
[junit4] 2> 877074 INFO (Thread-1484) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58837 58837
[junit4] 2> 877075 WARN (Thread-1484) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/live_nodes
[junit4] 2> 4 /solr/collections
[junit4] 2> 2 /solr/overseer/queue
[junit4] 2> 2 /solr/overseer/collection-queue-work
[junit4] 2> 2 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> 877077 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testOverseerFailure
[junit4] 2> 877077 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820801340833797-127.0.0.1:58837_solr-n_0000000001) closing
[junit4] 2> 877077 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:58837_solr
[junit4] 2> 877078 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 877081 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testPlaceholders
[junit4] 2> 877082 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 877082 INFO (Thread-1485) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 877082 INFO (Thread-1485) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 877220 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer start zk server on port:58875
[junit4] 2> 877220 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 877222 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 877227 INFO (zkCallback-1370-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1a5a12fc name:ZooKeeperConnection Watcher:127.0.0.1:58875/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 877227 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 877227 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 877227 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 877229 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 877230 INFO (zkCallback-1371-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@9ed7110 name:ZooKeeperConnection Watcher:127.0.0.1:58875 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 877230 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 877230 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 877234 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 877237 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 877240 INFO (zkCallback-1372-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@797e6efb name:ZooKeeperConnection Watcher:127.0.0.1:58875 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 877240 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 877240 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 877240 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 877246 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 877247 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 877250 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 877251 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 877254 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 877256 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 877258 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 877260 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 877261 INFO (zkCallback-1373-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2e57ec46 name:ZooKeeperConnection Watcher:127.0.0.1:58875/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 877263 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 877263 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 877264 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 877268 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 877270 INFO (zkCallback-1370-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 877270 INFO (zkCallback-1373-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 877270 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 877272 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 877274 INFO (zkCallback-1374-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ab97940 name:ZooKeeperConnection Watcher:127.0.0.1:58875/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 877274 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 877274 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 877274 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 877279 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
[junit4] 2> 877281 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
[junit4] 2> 877282 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 877284 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94820801965457412-127.0.0.1:58875_solr-n_0000000000
[junit4] 2> 877284 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58875_solr
[junit4] 2> 877284 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 877286 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820801965457412-127.0.0.1:58875_solr-n_0000000000) starting
[junit4] 2> 877288 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
[junit4] 2> 877291 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
[junit4] 2> 877293 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
[junit4] 2> 877296 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
[junit4] 2> 877299 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
[junit4] 2> 877302 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
[junit4] 2> 877306 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 877308 INFO (OverseerCollectionConfigSetProcessor-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 877308 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 877308 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 877313 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"12",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 0
[junit4] 2> 877313 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=12 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"12",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 877313 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
[junit4] 2> 877313 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 877315 INFO (zkCallback-1370-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 877315 INFO (zkCallback-1374-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 877315 INFO (zkCallback-1373-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 878028 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 878030 INFO (zkCallback-1370-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 878030 INFO (zkCallback-1373-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 878030 INFO (zkCallback-1374-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 878034 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard2/election/94820801965457411-node1_core1-n_0000000000
[junit4] 2> 878035 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard2
[junit4] 2> 878040 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 878042 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "core":"core1"} current state version: 1
[junit4] 2> 878044 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "core":"core1"}
[junit4] 2> 878047 INFO (zkCallback-1370-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 878047 INFO (zkCallback-1373-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 880187 INFO (OverseerCollectionConfigSetProcessor-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 880187 INFO (OverseerCollectionConfigSetProcessor-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=94820801965457412-127.0.0.1:58875_solr-n_0000000000) am no longer a leader.
[junit4] 2> 880204 WARN (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.Overseer Solr cannot talk to ZK, exiting Overseer main queue loop
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:336)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:333)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:333)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:310)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:289)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.firstElement(DistributedQueue.java:358)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:165)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:233)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 880204 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:58875_solr
[junit4] 2> 880205 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 880206 WARN (zkCallback-1373-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 880209 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58875 58875
[junit4] 2> 883448 INFO (Thread-1485) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58875 58875
[junit4] 2> 883451 WARN (Thread-1485) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/collections
[junit4] 2>
[junit4] 2> 883451 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testPlaceholders
[junit4] 2> 883451 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820801965457412-127.0.0.1:58875_solr-n_0000000000) closing
[junit4] 2> 883452 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testExternalClusterStateChangeBehavior
[junit4] 2> 883454 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 883455 INFO (Thread-1486) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 883455 INFO (Thread-1486) [ ]
[...truncated too long message...]
INFO (OverseerStateUpdate-94820806997573635-127.0.0.1:59314_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core2",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 953980 INFO (zkCallback-1451-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 953980 INFO (zkCallback-1454-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 953981 INFO (OverseerStateUpdate-94820806997573635-127.0.0.1:59314_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core3",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"} current state version: 2
[junit4] 2> 953981 INFO (OverseerStateUpdate-94820806997573635-127.0.0.1:59314_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core3",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 954052 INFO (zkCallback-1454-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 954052 INFO (zkCallback-1451-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 954116 INFO (TEST-OverseerTest.testReplay-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59314 59314
[junit4] 2> 955353 INFO (OverseerCollectionConfigSetProcessor-94820806997573635-127.0.0.1:59314_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 955353 INFO (OverseerCollectionConfigSetProcessor-94820806997573635-127.0.0.1:59314_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=94820806997573635-127.0.0.1:59314_solr-n_0000000000) am no longer a leader.
[junit4] 2> 955665 INFO (Thread-1494) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59314 59314
[junit4] 2> 955665 WARN (Thread-1494) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/live_nodes
[junit4] 2> 2 /solr/collections
[junit4] 2>
[junit4] 2> 955665 INFO (TEST-OverseerTest.testReplay-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testReplay
[junit4] 2> 955665 INFO (TEST-OverseerTest.testReplay-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820806997573635-127.0.0.1:59314_solr-n_0000000000) closing
[junit4] 2> 955665 INFO (OverseerStateUpdate-94820806997573635-127.0.0.1:59314_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:59314_solr
[junit4] 2> 955667 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 957731 INFO (SUITE-OverseerTest-seed#[A8631044D0ECD97]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.OverseerTest_A8631044D0ECD97-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene60): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=es_PE, timezone=SystemV/YST9YDT
[junit4] 2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=3,threads=1,free=36174160,total=231890944
[junit4] 2> NOTE: All tests run in this JVM: [TestLRUStatsCache, TestCollapseQParserPlugin, TestDistribDocBasedVersion, TestMinMaxOnMultiValuedField, TestCloudPivotFacet, GraphQueryTest, TestCustomSort, HttpPartitionTest, TestCrossCoreJoin, DOMUtilTest, TestSchemaResource, TestManagedStopFilterFactory, TestRestManager, TestManagedSchema, TestExtendedDismaxParser, RegexBoostProcessorTest, StandardRequestHandlerTest, PreAnalyzedFieldTest, ActionThrottleTest, PolyFieldTest, AsyncMigrateRouteKeyTest, DirectUpdateHandlerTest, TestReloadDeadlock, TestMergePolicyConfig, HighlighterTest, TestDynamicLoading, TestJmxMonitoredMap, JSONWriterTest, AliasIntegrationTest, RankQueryTest, VersionInfoTest, TestExpandComponent, UpdateParamsTest, SpatialRPTFieldTypeTest, TestCloudManagedSchemaConcurrent, LeaderElectionTest, TestLeaderElectionZkExpiry, HardAutoCommitTest, HdfsChaosMonkeySafeLeaderTest, CdcrUpdateLogTest, BadCopyFieldTest, JavabinLoaderTest, TestSolrCloudWithKerberosAlt, CloneFieldUpdateProcessorFactoryTest, TestReqParamsAPI, DocValuesTest, TestManagedSchemaFieldTypeResource, SolrRequestParserTest, ClusterStateTest, TestConfigSets, TestStressReorder, TestCloudSchemaless, SharedFSAutoReplicaFailoverUtilsTest, TestArbitraryIndexDir, TestConfigSetsAPIExclusivity, CdcrReplicationDistributedZkTest, TestXIncludeConfig, SampleTest, DocumentBuilderTest, UUIDUpdateProcessorFallbackTest, PingRequestHandlerTest, OverseerTest]
[junit4] Completed [203/555] on J0 in 97.74s, 12 tests, 1 failure, 1 skipped <<< FAILURES!
[...truncated 1142 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:775: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:719: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:59: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:233: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:516: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1432: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:987: There were test failures: 555 suites (8 ignored), 2200 tests, 1 failure, 115 ignored (77 assumptions) [seed: A8631044D0ECD97]
Total time: 71 minutes 9 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any