You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2015/11/25 08:57:32 UTC
[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 3802 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/3802/
1 tests failed.
FAILED: org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior
Error Message:
Illegal state, was: down expected:active clusterState:live nodes:[]collections:{c1=DocCollection(c1)={ "router":{"name":"implicit"}, "shards":{"shard1":{ "parent":null, "range":null, "state":"active", "replicas":{"core_node1":{ "base_url":"http://127.0.0.1/solr", "node_name":"node1", "core":"core1", "roles":"", "state":"down"}}}}}, test=LazyCollectionRef(test)}
Stack Trace:
java.lang.AssertionError: Illegal state, was: down expected:active clusterState:live nodes:[]collections:{c1=DocCollection(c1)={
"router":{"name":"implicit"},
"shards":{"shard1":{
"parent":null,
"range":null,
"state":"active",
"replicas":{"core_node1":{
"base_url":"http://127.0.0.1/solr",
"node_name":"node1",
"core":"core1",
"roles":"",
"state":"down"}}}}}, test=LazyCollectionRef(test)}
at __randomizedtesting.SeedInfo.seed([45ACCE11168CB533:2DB2CDFDF41CEF7D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractDistribZkTestBase.verifyReplicaStatus(AbstractDistribZkTestBase.java:229)
at org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior(OverseerTest.java:1247)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 10481 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J2/temp/solr.cloud.OverseerTest_45ACCE11168CB533-001/init-core-data-001
[junit4] 2> 966595 INFO (SUITE-OverseerTest-seed#[45ACCE11168CB533]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true)
[junit4] 2> 966595 INFO (SUITE-OverseerTest-seed#[45ACCE11168CB533]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore
[junit4] 2> 966595 INFO (SUITE-OverseerTest-seed#[45ACCE11168CB533]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore end
[junit4] 2> 966599 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testPlaceholders
[junit4] 2> 966599 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 966599 INFO (Thread-2276) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 966599 INFO (Thread-2276) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 966699 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.ZkTestServer start zk server on port:37394
[junit4] 2> 966699 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 966700 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 966702 INFO (zkCallback-688-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@ffc436b name:ZooKeeperConnection Watcher:127.0.0.1:37394/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 966702 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 966703 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 966703 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 966703 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 966704 INFO (zkCallback-689-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@53d9ebf2 name:ZooKeeperConnection Watcher:127.0.0.1:37394 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 966704 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 966704 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 966705 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 966706 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 966706 INFO (zkCallback-690-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@483870ed name:ZooKeeperConnection Watcher:127.0.0.1:37394 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 966707 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 966707 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 966707 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 966710 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 966711 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 966712 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 966713 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 966714 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 966715 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 966717 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 966717 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 966718 INFO (zkCallback-691-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5b0271cf name:ZooKeeperConnection Watcher:127.0.0.1:37394/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 966718 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 966718 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 966720 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 966721 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 966722 INFO (zkCallback-688-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 966722 INFO (zkCallback-691-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 966723 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 966724 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 966726 INFO (zkCallback-692-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2f601027 name:ZooKeeperConnection Watcher:127.0.0.1:37394/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 966726 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 966726 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 966726 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 966731 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
[junit4] 2> 966733 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
[junit4] 2> 966735 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 966735 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94924762049085444-127.0.0.1:37394_solr-n_0000000000
[junit4] 2> 966736 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37394_solr
[junit4] 2> 966736 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 966738 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.Overseer Overseer (id=94924762049085444-127.0.0.1:37394_solr-n_0000000000) starting
[junit4] 2> 966739 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
[junit4] 2> 966741 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
[junit4] 2> 966743 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
[junit4] 2> 966754 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
[junit4] 2> 966761 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
[junit4] 2> 966763 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
[junit4] 2> 966768 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 966768 INFO (OverseerCollectionConfigSetProcessor-94924762049085444-127.0.0.1:37394_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 966769 INFO (OverseerStateUpdate-94924762049085444-127.0.0.1:37394_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 966770 INFO (OverseerStateUpdate-94924762049085444-127.0.0.1:37394_solr-n_0000000000) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 966772 INFO (OverseerStateUpdate-94924762049085444-127.0.0.1:37394_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"12",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 0
[junit4] 2> 966773 INFO (OverseerStateUpdate-94924762049085444-127.0.0.1:37394_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=12 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"12",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 966773 INFO (OverseerStateUpdate-94924762049085444-127.0.0.1:37394_solr-n_0000000000) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
[junit4] 2> 966773 INFO (OverseerStateUpdate-94924762049085444-127.0.0.1:37394_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 966774 INFO (zkCallback-692-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 966774 INFO (zkCallback-688-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 966774 INFO (zkCallback-691-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 967270 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 967271 INFO (zkCallback-691-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 967271 INFO (zkCallback-692-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 967271 INFO (zkCallback-688-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 967276 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94924762049085443-node1_core1-n_0000000000
[junit4] 2> 967277 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard1
[junit4] 2> 967279 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 967282 INFO (OverseerStateUpdate-94924762049085444-127.0.0.1:37394_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 967285 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1513d9172530004, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 967287 INFO (zkCallback-688-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 967287 INFO (zkCallback-691-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 967287 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1513d9172530003, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 968770 INFO (OverseerCollectionConfigSetProcessor-94924762049085444-127.0.0.1:37394_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 968771 INFO (OverseerCollectionConfigSetProcessor-94924762049085444-127.0.0.1:37394_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=94924762049085444-127.0.0.1:37394_solr-n_0000000000) am no longer a leader.
[junit4] 2> 968785 WARN (OverseerStateUpdate-94924762049085444-127.0.0.1:37394_solr-n_0000000000) [ ] o.a.s.c.Overseer Solr cannot talk to ZK, exiting Overseer main queue loop
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue/qn-0000000001
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:244)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:241)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:241)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.removeFirst(DistributedQueue.java:386)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.poll(DistributedQueue.java:188)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:229)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 968785 INFO (OverseerStateUpdate-94924762049085444-127.0.0.1:37394_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37394_solr
[junit4] 2> 968786 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 968787 WARN (zkCallback-691-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 968788 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37394 37394
[junit4] 2> 968984 INFO (Thread-2276) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37394 37394
[junit4] 2> 968985 WARN (Thread-2276) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/collections
[junit4] 2>
[junit4] 2> 968986 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testPlaceholders
[junit4] 2> 968986 INFO (TEST-OverseerTest.testPlaceholders-seed#[45ACCE11168CB533]) [ ] o.a.s.c.Overseer Overseer (id=94924762049085444-127.0.0.1:37394_solr-n_0000000000) closing
[junit4] 2> 968991 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testStateChange
[junit4] 2> 968991 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 968991 INFO (Thread-2277) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 968992 INFO (Thread-2277) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 969091 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.ZkTestServer start zk server on port:55698
[junit4] 2> 969092 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 969093 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 969096 INFO (zkCallback-694-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3dd2d94d name:ZooKeeperConnection Watcher:127.0.0.1:55698/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 969096 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 969096 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 969096 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 969097 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 969098 INFO (zkCallback-695-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@28eed037 name:ZooKeeperConnection Watcher:127.0.0.1:55698 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 969098 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 969098 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 969099 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 969100 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 969101 INFO (zkCallback-696-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@393c43f3 name:ZooKeeperConnection Watcher:127.0.0.1:55698 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 969102 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 969102 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 969102 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 969106 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 969107 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 969109 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 969111 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 969112 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 969112 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 969114 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 969114 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 969116 INFO (zkCallback-697-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@43c3faa0 name:ZooKeeperConnection Watcher:127.0.0.1:55698/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 969116 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 969116 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 969116 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 969118 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
[junit4] 2> 969119 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
[junit4] 2> 969123 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 969124 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94924762205913091-127.0.0.1:55698_solr-n_0000000000
[junit4] 2> 969124 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:55698_solr
[junit4] 2> 969124 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 969125 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.Overseer Overseer (id=94924762205913091-127.0.0.1:55698_solr-n_0000000000) starting
[junit4] 2> 969126 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
[junit4] 2> 969128 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
[junit4] 2> 969130 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
[junit4] 2> 969133 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
[junit4] 2> 969135 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
[junit4] 2> 969137 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
[junit4] 2> 969141 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 969142 INFO (OverseerCollectionConfigSetProcessor-94924762205913091-127.0.0.1:55698_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 969142 INFO (OverseerStateUpdate-94924762205913091-127.0.0.1:55698_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 969142 INFO (OverseerStateUpdate-94924762205913091-127.0.0.1:55698_solr-n_0000000000) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 969146 INFO (OverseerStateUpdate-94924762205913091-127.0.0.1:55698_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"} current state version: 0
[junit4] 2> 969147 INFO (OverseerStateUpdate-94924762205913091-127.0.0.1:55698_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 969147 INFO (OverseerStateUpdate-94924762205913091-127.0.0.1:55698_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 969149 INFO (zkCallback-697-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 969149 INFO (zkCallback-694-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 969197 INFO (OverseerStateUpdate-94924762205913091-127.0.0.1:55698_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 969198 INFO (OverseerStateUpdate-94924762205913091-127.0.0.1:55698_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"active"}
[junit4] 2> 969198 INFO (OverseerStateUpdate-94924762205913091-127.0.0.1:55698_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator node=core_node1 is already registered
[junit4] 2> 969198 INFO (OverseerStateUpdate-94924762205913091-127.0.0.1:55698_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 969302 INFO (zkCallback-694-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 969302 INFO (zkCallback-697-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 969349 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:55698 55698
[junit4] 2> 969539 INFO (Thread-2277) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:55698 55698
[junit4] 2> 969540 WARN (Thread-2277) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/live_nodes
[junit4] 2> 2 /solr/collections
[junit4] 2>
[junit4] 2> 969541 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testStateChange
[junit4] 2> 969541 INFO (TEST-OverseerTest.testStateChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.Overseer Overseer (id=94924762205913091-127.0.0.1:55698_solr-n_0000000000) closing
[junit4] 2> 969541 INFO (OverseerStateUpdate-94924762205913091-127.0.0.1:55698_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:55698_solr
[junit4] 2> 969542 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 969546 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testShardLeaderChange
[junit4] 2> 969547 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 969547 INFO (Thread-2278) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 969547 INFO (Thread-2278) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 969647 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.ZkTestServer start zk server on port:59523
[junit4] 2> 969647 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 969648 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 969651 INFO (zkCallback-699-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6c2d5c65 name:ZooKeeperConnection Watcher:127.0.0.1:59523/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 969651 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 969651 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 969651 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 969652 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 969653 INFO (zkCallback-700-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@15e2da62 name:ZooKeeperConnection Watcher:127.0.0.1:59523 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 969653 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 969653 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 969656 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 969656 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 969657 INFO (zkCallback-701-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6734186f name:ZooKeeperConnection Watcher:127.0.0.1:59523 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 969657 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 969658 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 969658 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 969661 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 969662 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 969663 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 969664 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 969666 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 969668 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 969668 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 969670 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 969671 INFO (Thread-2279) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 969671 INFO (zkCallback-702-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3f08de20 name:ZooKeeperConnection Watcher:127.0.0.1:59523/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 969671 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 969672 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 969673 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 969674 INFO (zkCallback-703-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@634bb9b9 name:ZooKeeperConnection Watcher:127.0.0.1:59523/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 969674 INFO (Thread-2279) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 969674 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 969674 INFO (Thread-2279) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 969676 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
[junit4] 2> 969678 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 969680 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
[junit4] 2> 969681 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
[junit4] 2> 969682 INFO (zkCallback-702-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 969683 INFO (Thread-2279) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 969684 INFO (Thread-2279) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94924762242220036-127.0.0.1:59523_solr-n_0000000000
[junit4] 2> 969684 INFO (Thread-2279) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59523_solr
[junit4] 2> 969684 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 969686 INFO (Thread-2279) [ ] o.a.s.c.Overseer Overseer (id=94924762242220036-127.0.0.1:59523_solr-n_0000000000) starting
[junit4] 2> 969690 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
[junit4] 2> 969691 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
[junit4] 2> 969693 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
[junit4] 2> 969696 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
[junit4] 2> 969699 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
[junit4] 2> 969702 INFO (Thread-2279) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 969703 INFO (OverseerCollectionConfigSetProcessor-94924762242220036-127.0.0.1:59523_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 969704 INFO (Thread-2279) [ ] o.a.s.SolrTestCaseJ4 Killing overseer.
[junit4] 2> 969704 INFO (OverseerStateUpdate-94924762242220036-127.0.0.1:59523_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 969705 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 969705 INFO (Thread-2279) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 969708 INFO (zkCallback-705-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@72cd694 name:ZooKeeperConnection Watcher:127.0.0.1:59523/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 969708 INFO (Thread-2279) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 969708 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 969709 INFO (Thread-2279) [ ] o.a.s.c.Overseer Overseer (id=94924762242220036-127.0.0.1:59523_solr-n_0000000000) closing
[junit4] 2> 969709 INFO (OverseerStateUpdate-94924762242220036-127.0.0.1:59523_solr-n_0000000000) [ ] o.a.s.c.Overseer According to ZK I (id=94924762242220036-127.0.0.1:59523_solr-n_0000000000) am no longer a leader.
[junit4] 2> 969709 INFO (Thread-2279) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 969709 INFO (OverseerStateUpdate-94924762242220036-127.0.0.1:59523_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:59523_solr
[junit4] 2> 969716 ERROR (OverseerHdfsCoreFailoverThread-94924762242220036-127.0.0.1:59523_solr-n_0000000000) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread OverseerAutoReplicaFailoverThread had an error in its thread work loop.:org.apache.solr.common.SolrException: Error reading cluster properties
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getClusterProps(ZkStateReader.java:737)
[junit4] 2> at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.doWork(OverseerAutoReplicaFailoverThread.java:152)
[junit4] 2> at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:131)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getClusterProps(ZkStateReader.java:730)
[junit4] 2> ... 3 more
[junit4] 2>
[junit4] 2> 969716 ERROR (OverseerCollectionConfigSetProcessor-94924762242220036-127.0.0.1:59523_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Unable to prioritize overseer
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.cloud.OverseerNodePrioritizer.prioritizeOverseerNodes(OverseerNodePrioritizer.java:59)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:159)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 969719 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 969720 INFO (Thread-2279) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 969721 INFO (Thread-2279) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94924762242220037-127.0.0.1:59523_solr-n_0000000001
[junit4] 2> 969721 INFO (Thread-2279) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59523_solr
[junit4] 2> 969721 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 969723 INFO (Thread-2279) [ ] o.a.s.c.Overseer Overseer (id=94924762242220037-127.0.0.1:59523_solr-n_0000000001) starting
[junit4] 2> 969729 INFO (Thread-2279) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 969730 INFO (OverseerCollectionConfigSetProcessor-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 969730 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 969731 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 969735 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 0
[junit4] 2> 969735 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 969735 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
[junit4] 2> 969735 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 969737 INFO (zkCallback-702-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 969737 INFO (zkCallback-705-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 970184 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 970185 INFO (zkCallback-705-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 970185 INFO (zkCallback-702-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 970188 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94924762242220035-node1_core1-n_0000000000
[junit4] 2> 970189 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard1
[junit4] 2> 970191 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 970194 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 970196 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 1
[junit4] 2> 970196 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 970196 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 970196 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 970200 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94924762242220035-node1_core1-n_0000000001
[junit4] 2> 970200 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/94924762242220035-node1_core1-n_0000000001 , seq(0) /collections/collection1/leader_elect/shard1/election/94924762242220035-node1_core1-n_0000000000
[junit4] 2> 970201 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 970201 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 970202 INFO (zkCallback-707-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@29f5b43c name:ZooKeeperConnection Watcher:127.0.0.1:59523/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 970202 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 970203 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 970204 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 970206 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node2
[junit4] 2> 970207 INFO (zkCallback-705-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 970207 INFO (zkCallback-707-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 970207 INFO (zkCallback-702-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 970210 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 970210 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 1
[junit4] 2> 970211 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 970211 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 970213 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94924762242220035-node1_core1-n_0000000002
[junit4] 2> 970214 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/94924762242220035-node1_core1-n_0000000002 , seq(0) /collections/collection1/leader_elect/shard1/election/94924762242220035-node1_core1-n_0000000000
[junit4] 2> 970214 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 970313 INFO (zkCallback-705-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 970313 INFO (zkCallback-702-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 970313 INFO (zkCallback-699-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 970313 INFO (zkCallback-707-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 970418 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"} current state version: 2
[junit4] 2> 970418 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"}
[junit4] 2> 970418 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=1
[junit4] 2> 970419 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 970521 INFO (zkCallback-702-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 970521 INFO (zkCallback-699-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 970521 INFO (zkCallback-705-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 970521 INFO (zkCallback-707-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 970917 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 970920 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94924762242220038-node2_core4-n_0000000003
[junit4] 2> 970921 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/94924762242220035-node1_core1-n_0000000002 to know if I could be the leader
[junit4] 2> 970922 INFO (zkCallback-705-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 970922 INFO (zkCallback-702-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 970922 INFO (zkCallback-699-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 970923 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1513d917dd60003, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 970923 INFO (zkCallback-707-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 970926 INFO (zkCallback-707-thread-2) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 970930 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node2/solr/",
[junit4] 2> "core":"core4",
[junit4] 2> "state":"active"} current state version: 3
[junit4] 2> 971032 INFO (zkCallback-707-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 971033 INFO (zkCallback-699-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 971034 INFO (zkCallback-705-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 971126 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 971127 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 971128 INFO (zkCallback-708-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@56304c8b name:ZooKeeperConnection Watcher:127.0.0.1:59523/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 971128 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 971128 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 971130 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 971131 INFO (Thread-2279) [ ] o.a.s.SolrTestCaseJ4 Killing overseer.
[junit4] 2> 971131 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 971132 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 971132 INFO (Thread-2279) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 971133 INFO (zkCallback-707-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 971133 INFO (zkCallback-699-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 971133 INFO (zkCallback-708-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 971135 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 971135 INFO (zkCallback-709-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@686a9092 name:ZooKeeperConnection Watcher:127.0.0.1:59523/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 971135 INFO (Thread-2279) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 971135 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 971135 INFO (Thread-2279) [ ] o.a.s.c.Overseer Overseer (id=94924762242220037-127.0.0.1:59523_solr-n_0000000001) closing
[junit4] 2> 971135 INFO (OverseerStateUpdate-94924762242220037-127.0.0.1:59523_solr-n_0000000001) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:59523_solr
[junit4] 2> 971135 INFO (Thread-2279) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 971137 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 971139 INFO (Thread-2279) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 971140 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94924762242220039-node1_core1-n_0000000004
[junit4] 2> 971140 INFO (Thread-2279) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94924762242220040-127.0.0.1:59523_solr-n_0000000002
[junit4] 2> 971140 INFO (Thread-2279) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59523_solr
[junit4] 2> 971140 INFO (Thread-2279) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 971140 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.LeaderElector Watching path /collections/collection1/leader_elect/shard1/election/94924762242220038-node2_core4-n_0000000003 to know if I could be the leader
[junit4] 2> 971141 INFO (zkCallback-708-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 971141 INFO (zkCallback-707-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 971141 INFO (zkCallback-699-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 971143 INFO (zkCallback-708-thread-2) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 971143 INFO (Thread-2279) [ ] o.a.s.c.Overseer Overseer (id=94924762242220040-127.0.0.1:59523_solr-n_0000000002) starting
[junit4] 2> 971149 INFO (Thread-2279) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 971149 INFO (OverseerCollectionConfigSetProcessor-94924762242220040-127.0.0.1:59523_solr-n_0000000002) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 971149 INFO (OverseerStateUpdate-94924762242220040-127.0.0.1:59523_solr-n_0000000002) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 971150 INFO (OverseerStateUpdate-94924762242220040-127.0.0.1:59523_solr-n_0000000002) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 971152 INFO (OverseerStateUpdate-94924762242220040-127.0.0.1:59523_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 4
[junit4] 2> 971152 INFO (OverseerStateUpdate-94924762242220040-127.0.0.1:59523_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 971152 INFO (OverseerStateUpdate-94924762242220040-127.0.0.1:59523_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 971153 INFO (zkCallback-699-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 971154 INFO (zkCallback-709-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 971154 INFO (zkCallback-708-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 971156 INFO (OverseerStateUpdate-94924762242220040-127.0.0.1:59523_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1",
[junit4] 2> "state":"active"} current state version: 5
[junit4] 2> 971258 INFO (zkCallback-709-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 971259 INFO (zkCallback-699-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 971259 INFO (zkCallback-708-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 972641 WARN (zkCallback-707-thread-2) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 972643 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 972643 INFO (OverseerStateUpdate-94924762242220040-127.0.0.1:59523_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 6
[junit4] 2> 972644 INFO (OverseerStateUpdate-94924762242220040-127.0.0.1:59523_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 972644 INFO (OverseerStateUpdate-94924762242220040-127.0.0.1:59523_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 972646 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94924762242220039-node1_core1-n_0000000005
[junit4] 2> 972646 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/94924762242220039-node1_core1-n_0000000005 , seq(0) /collections/collection1/leader_elect/shard1/election/94924762242220039-node1_core1-n_0000000004
[junit4] 2> 972647 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 972664 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 972668 INFO (zkCallback-711-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@58e34b5 name:ZooKeeperConnection Watcher:127.0.0.1:59523/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 972668 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 972668 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 972670 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 972671 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node2
[junit4] 2> 972672 INFO (zkCallback-699-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 972673 INFO (zkCallback-709-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 972673 INFO (zkCallback-708-thread-2) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 972673 INFO (zkCallback-711-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 972674 INFO (TEST-OverseerTest.testShardLeaderChange-seed#[45ACCE11168CB533]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 972676 INFO (OverseerStateUpdate-94924762242220040-127.0.0.1:59523_solr-n_0000000002) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 6
[junit4] 2> 972677 INFO (OverseerStateUpdate-94924762242220040-127.0.0.1:59523_solr-n_0000000002) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 972677 INFO
[...truncated too long message...]
> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1@631b65be rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@e8307aa[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 9]
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
[junit4] 2> at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:261)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2> 1017900 WARN (zkCallback-803-thread-2) [ ] o.a.s.c.LeaderElector
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:325)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:325)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:107)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:58)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:388)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:264)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1019399 WARN (zkCallback-803-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1019402 INFO (TEST-OverseerTest.testShardAssignment-seed#[45ACCE11168CB533]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:33625 33625
[junit4] 2> 1019666 INFO (Thread-2287) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:33625 33625
[junit4] 2> 1019667 WARN (Thread-2287) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/collections
[junit4] 2>
[junit4] 2> 1019668 INFO (TEST-OverseerTest.testShardAssignment-seed#[45ACCE11168CB533]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testShardAssignment
[junit4] 2> 1019668 INFO (TEST-OverseerTest.testShardAssignment-seed#[45ACCE11168CB533]) [ ] o.a.s.c.Overseer Overseer (id=94924765201235971-127.0.0.1:33625_solr-n_0000000000) closing
[junit4] 2> 1019668 INFO (OverseerStateUpdate-94924765201235971-127.0.0.1:33625_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:33625_solr
[junit4] 2> 1019669 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 1022669 INFO (SUITE-OverseerTest-seed#[45ACCE11168CB533]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J2/temp/solr.cloud.OverseerTest_45ACCE11168CB533-001
[junit4] 2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=8016, maxDocsPerChunk=10, blockSize=7), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=8016, blockSize=7)), sim=DefaultSimilarity, locale=pt, timezone=Africa/Johannesburg
[junit4] 2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_80 (64-bit)/cpus=4,threads=1,free=76441512,total=448266240
[junit4] 2> NOTE: All tests run in this JVM: [TestRebalanceLeaders, AddSchemaFieldsUpdateProcessorFactoryTest, SolrCoreCheckLockOnStartupTest, TestRestoreCore, StatelessScriptUpdateProcessorFactoryTest, DistributedIntervalFacetingTest, UnloadDistributedZkTest, SuggesterTSTTest, LeaderInitiatedRecoveryOnShardRestartTest, DocExpirationUpdateProcessorFactoryTest, BasicDistributedZk2Test, FieldMutatingUpdateProcessorTest, TestUtils, TestNoOpRegenerator, FullSolrCloudDistribCmdsTest, DistributedQueryComponentOptimizationTest, TestPostingsSolrHighlighter, TestFieldTypeCollectionResource, TestLeaderElectionZkExpiry, DocValuesMultiTest, TestJmxIntegration, TestRecovery, LukeRequestHandlerTest, OverseerStatusTest, TestSolrDeletionPolicy1, PluginInfoTest, TestReRankQParserPlugin, QueryParsingTest, CoreAdminRequestStatusTest, TestReversedWildcardFilterFactory, ForceLeaderTest, SearchHandlerTest, TestMiniSolrCloudCluster, TestIntervalFaceting, TestSchemaVersionResource, ExplicitHLLTest, HLLSerializationTest, TestChildDocTransformer, PrimitiveFieldTypeTest, TestPerFieldSimilarityWithDefaultOverride, DirectSolrConnectionTest, TestConfigSetImmutable, ScriptEngineTest, HdfsSyncSliceTest, CloneFieldUpdateProcessorFactoryTest, TestIndexSearcher, CircularListTest, TestReload, BadIndexSchemaTest, BadComponentTest, TestSort, NoCacheHeaderTest, HighlighterConfigTest, AtomicUpdatesTest, SparseHLLTest, MultiThreadedOCPTest, DirectUpdateHandlerOptimizeTest, SimpleCollectionCreateDeleteTest, TestInfoStreamLogging, CollectionTooManyReplicasTest, TestExactSharedStatsCache, DateRangeFieldTest, TestGroupingSearch, TestReloadDeadlock, TestTrackingShardHandlerFactory, TestFileDictionaryLookup, MergeStrategyTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestHashQParserPlugin, SimpleFacetsTest, RequestHandlersTest, TestFuzzyAnalyzedSuggestions, HardAutoCommitTest, TestStressRecovery, TestSolrIndexConfig, HdfsChaosMonkeySafeLeaderTest, TestCloudInspectUtil, TestCSVResponseWriter, RemoteQueryErrorTest, ConcurrentDeleteAndCreateCollectionTest, TestSolrXml, OverseerTest]
[junit4] Completed [225/551] on J2 in 56.10s, 12 tests, 1 failure, 1 skipped <<< FAILURES!
[...truncated 1032 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:785: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:729: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/common-build.xml:526: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/common-build.xml:1452: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/common-build.xml:1006: There were test failures: 551 suites (7 ignored), 2177 tests, 1 failure, 52 ignored (38 assumptions) [seed: 45ACCE11168CB533]
Total time: 60 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any