You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/03/01 23:33:34 UTC
[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_51) - Build # 9534
- Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9534/
Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseParallelGC -XX:-UseSuperWord
1 tests failed.
REGRESSION: org.apache.solr.cloud.OverseerTest.testOverseerFailure
Error Message:
KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
at __randomizedtesting.SeedInfo.seed([4BDED763554BF469:4FD6589047EE1B48]:0)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:429)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:426)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:154)
at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:590)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:744)
Build Log:
[...truncated 10764 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> 628382 T2554 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
[junit4] 2> 628382 T2554 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1393712582290
[junit4] 2> 628382 T2554 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 628385 T2554 oas.SolrTestCaseJ4.setUp ###Starting testReplay
[junit4] 2> 628385 T2554 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 628385 T2555 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 628485 T2554 oasc.ZkTestServer.run start zk server on port:33512
[junit4] 2> 628486 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 628589 T2561 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d26f603 name:ZooKeeperConnection Watcher:127.0.0.1:33512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 628589 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 628590 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 628592 T2563 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1919bf2d name:ZooKeeperConnection Watcher:127.0.0.1:33512 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 628593 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 628594 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 628596 T2565 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@454a8440 name:ZooKeeperConnection Watcher:127.0.0.1:33512 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 628596 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 628596 T2554 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 628598 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 628600 T2554 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 628601 T2554 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 628602 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 628605 T2554 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 628607 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 628608 T2567 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@555f5cb6 name:ZooKeeperConnection Watcher:127.0.0.1:33512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 628609 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 628609 T2554 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 628613 T2554 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 628615 T2554 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:33512_solr
[junit4] 2> 628615 T2554 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 628616 T2554 oasc.Overseer.start Overseer (id=91338347793285123-127.0.0.1:33512_solr-n_0000000000) starting
[junit4] 2> 628619 T2554 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 628625 T2554 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 628627 T2569 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 628627 T2569 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 628628 T2568 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 628629 T2568 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
[junit4] 2> 628629 T2568 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 628630 T2561 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 628633 T2568 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core2",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 628633 T2561 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 628635 T2568 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 628636 T2568 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 628637 T2568 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core3",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 628638 T2567 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 628639 T2561 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 628735 T2554 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:33512 33512
[junit4] 2> 628829 T2554 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 628833 T2554 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
[junit4] 2> 628834 T2554 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 628834 T2570 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 628934 T2554 oasc.ZkTestServer.run start zk server on port:52276
[junit4] 2> 628935 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 628939 T2576 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@62d3d795 name:ZooKeeperConnection Watcher:127.0.0.1:52276/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 628940 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 628941 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 628943 T2578 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@277d0339 name:ZooKeeperConnection Watcher:127.0.0.1:52276 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 628943 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 628966 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 628968 T2580 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@300fe69b name:ZooKeeperConnection Watcher:127.0.0.1:52276 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 628969 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 628969 T2554 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 628973 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 628977 T2554 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 628979 T2554 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 628982 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 628987 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 628989 T2582 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1388a2eb name:ZooKeeperConnection Watcher:127.0.0.1:52276/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 628989 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 628991 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 628996 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 629000 T2582 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 629000 T2576 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 629001 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 629003 T2584 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47280f07 name:ZooKeeperConnection Watcher:127.0.0.1:52276/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 629004 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 629005 T2554 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 629008 T2554 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 629014 T2554 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:52276_solr
[junit4] 2> 629015 T2554 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 629018 T2554 oasc.Overseer.start Overseer (id=91338347822776324-127.0.0.1:52276_solr-n_0000000000) starting
[junit4] 2> 629021 T2554 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 629024 T2554 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 629028 T2554 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 629031 T2586 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 629032 T2586 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 629032 T2585 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 630034 T2584 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 630036 T2585 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 630037 T2585 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 630037 T2585 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 630038 T2585 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 630040 T2584 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 630041 T2582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 630041 T2576 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 630535 T2554 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 630552 T2554 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 630559 T2584 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 630560 T2554 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 630561 T2585 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 630564 T2554 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 630564 T2584 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 630567 T2585 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 630567 T2585 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 630571 T2584 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 630674 T2576 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 630674 T2582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 631729 T2554 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 631738 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 631740 T2588 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3aefbc7a name:ZooKeeperConnection Watcher:127.0.0.1:52276/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 631740 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 631743 T2554 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:52276_solr
[junit4] 2> 631743 T2554 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 631745 T2554 oasc.Overseer.start Overseer (id=91338347822776325-127.0.0.1:52276_solr-n_0000000001) starting
[junit4] 2> 631751 T2590 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 631752 T2590 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 631752 T2589 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 631755 T2589 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 631755 T2589 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 631756 T2589 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 631759 T2588 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 631760 T2582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 631760 T2576 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 631813 T2588 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 631813 T2554 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 631814 T2589 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 631817 T2554 oasc.LeaderElector.checkIfIamLeader WARN Failed setting watch org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/91338347822776323-node1_core1-n_0000000002
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:271)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:271)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
[junit4] 2> at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:154)
[junit4] 2> at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:590)
[junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] 2> at java.lang.reflect.Method.invoke(Method.java:606)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4] 2> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 631818 T2589 oasc.SolrException.log ERROR Problem cleaning up collection in zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:171)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:171)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:633)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:628)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:628)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:628)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:949)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:235)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:192)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 631819 T2582 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 631820 T2554 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 631820 T2588 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 631823 T2588 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 631823 T2576 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 631824 T2589 oasc.Overseer$ClusterStateUpdater.setShardLeader ERROR Could not mark shard leader for non existing collection:collection1
[junit4] 2> 631824 T2589 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue-work/qn-
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:240)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:237)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:237)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.createData(DistributedQueue.java:284)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:271)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:193)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 631826 T2554 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52276 52276
[junit4] 2> 632044 T2554 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=4BDED763554BF469 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=th_TH_TH_#u-nu-thai -Dtests.timezone=America/Guatemala -Dtests.file.encoding=US-ASCII
[junit4] ERROR 3.23s J1 | OverseerTest.testOverseerFailure <<<
[junit4] > Throwable #1: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
[junit4] > at __randomizedtesting.SeedInfo.seed([4BDED763554BF469:4FD6589047EE1B48]:0)
[junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
[junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:429)
[junit4] > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:426)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
[junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
[junit4] > at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
[junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
[junit4] > at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:154)
[junit4] > at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:590)
[junit4] > at java.lang.Thread.run(Thread.java:744)
[junit4] 2> 632065 T2554 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
[junit4] 2> 632065 T2554 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 632066 T2591 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 632166 T2554 oasc.ZkTestServer.run start zk server on port:47970
[junit4] 2> 632167 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632172 T2597 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e1b9df1 name:ZooKeeperConnection Watcher:127.0.0.1:47970/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632173 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632174 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632176 T2599 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a2088df name:ZooKeeperConnection Watcher:127.0.0.1:47970 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632176 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632203 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632205 T2601 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@273a8ef name:ZooKeeperConnection Watcher:127.0.0.1:47970 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632205 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632206 T2554 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 632211 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 632215 T2554 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 632218 T2554 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 632221 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632225 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632228 T2603 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@799f9960 name:ZooKeeperConnection Watcher:127.0.0.1:47970/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632228 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632230 T2554 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 632233 T2554 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 632238 T2554 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:47970_solr
[junit4] 2> 632238 T2554 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 632240 T2554 oasc.Overseer.start Overseer (id=91338348035244035-127.0.0.1:47970_solr-n_0000000000) starting
[junit4] 2> 632242 T2554 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 632244 T2554 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 632247 T2554 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 632249 T2605 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 632249 T2605 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 632249 T2604 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 632250 T2554 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 632257 T2604 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 632258 T2604 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 632259 T2604 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 632262 T2603 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 632263 T2597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 632309 T2554 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 632311 T2603 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 632314 T2604 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 632314 T2604 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"active"}
[junit4] 2> 632315 T2604 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
[junit4] 2> 632315 T2604 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 632319 T2603 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 632421 T2597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 632467 T2554 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:47970 47970
[junit4] 2> 632652 T2554 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 632660 T2554 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
[junit4] 2> 632661 T2554 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 632662 T2606 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 632761 T2554 oasc.ZkTestServer.run start zk server on port:41353
[junit4] 2> 632763 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632767 T2612 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ff22f1a name:ZooKeeperConnection Watcher:127.0.0.1:41353 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632767 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632785 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632787 T2614 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73814971 name:ZooKeeperConnection Watcher:127.0.0.1:41353 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632788 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632788 T2554 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 632794 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632796 T2616 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42630ecc name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632796 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632797 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 632800 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632802 T2618 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10c3cc44 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632802 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632804 T2554 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 632808 T2554 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 632814 T2554 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:41353_solr
[junit4] 2> 632814 T2554 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 632818 T2554 oasc.Overseer.start Overseer (id=91338348074041347-127.0.0.1:41353_solr-n_0000000000) starting
[junit4] 2> 632821 T2554 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 632825 T2554 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 632831 T2554 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 632834 T2620 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 632835 T2554 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 632835 T2620 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 632835 T2619 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 632838 T2554 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 632840 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632843 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632845 T2622 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e85a09a name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632845 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632847 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632850 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
[junit4] 2> 632854 T2622 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 632854 T2616 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 632855 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632858 T2624 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4bad82ba name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632858 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632860 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632864 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 632867 T2622 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 632868 T2616 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 632868 T2624 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 632868 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632870 T2626 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f184eb0 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632871 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632873 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632876 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 632880 T2616 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 632880 T2624 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 632880 T2626 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 632881 T2622 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 632881 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632883 T2628 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e7d0c89 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632883 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632886 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632890 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
[junit4] 2> 632893 T2622 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 632893 T2616 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 632893 T2624 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 632893 T2628 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 632893 T2626 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 632894 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632896 T2630 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d620304 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632897 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632899 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632902 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
[junit4] 2> 632906 T2626 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 632906 T2616 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 632906 T2622 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 632908 T2624 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 632907 T2628 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 632908 T2630 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 632908 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632910 T2632 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e744411 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632911 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632913 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632916 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
[junit4] 2> 632920 T2616 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 632920 T2622 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 632920 T2624 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 632920 T2626 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 632921 T2632 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 632921 T2630 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 632921 T2628 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 632922 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632924 T2634 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31d9d88d name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632924 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632926 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632929 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
[junit4] 2> 632933 T2624 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 632934 T2630 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 632934 T2632 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 632934 T2634 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 632934 T2628 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 632934 T2622 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 632934 T2616 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 632934 T2626 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 632935 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632937 T2636 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b01fdf1 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632937 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632938 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632940 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
[junit4] 2> 632942 T2616 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 632942 T2628 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 632942 T2622 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 632942 T2636 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 632942 T2632 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 632942 T2630 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 632942 T2626 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 632942 T2634 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 632942 T2624 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 632943 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632944 T2638 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71d30268 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632944 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632945 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632946 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
[junit4] 2> 632949 T2638 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 632949 T2624 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 632949 T2626 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 632949 T2634 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 632950 T2628 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 632949 T2622 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 632950 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632949 T2616 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 632949 T2630 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 632949 T2632 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 632949 T2636 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 632952 T2640 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ff6a067 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632952 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632954 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632957 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
[junit4] 2> 632961 T2622 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 632961 T2626 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 632961 T2628 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 632961 T2640 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 632961 T2634 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 632961 T2630 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 632961 T2624 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 632961 T2638 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 632961 T2632 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 632961 T2616 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 632962 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632964 T2642 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@400457f0 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632962 T2636 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 632964 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632966 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632969 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
[junit4] 2> 632974 T2632 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 632974 T2622 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 632975 T2630 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 632975 T2642 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 632974 T2626 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 632974 T2640 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 632975 T2638 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 632975 T2636 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 632975 T2628 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 632975 T2616 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 632975 T2634 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 632975 T2624 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 632976 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632978 T2644 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57f11c81 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632979 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632980 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632983 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
[junit4] 2> 632987 T2624 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 632987 T2636 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 632988 T2616 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 632988 T2630 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 632988 T2640 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 632987 T2628 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 632988 T2644 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 632988 T2638 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 632987 T2626 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 632987 T2632 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 632987 T2622 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 632989 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 632988 T2634 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 632988 T2642 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 632991 T2646 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@243c58a9 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 632992 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 632994 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 632998 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
[junit4] 2> 633004 T2632 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 633004 T2630 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 633004 T2638 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 633004 T2634 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 633004 T2644 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 633006 T2646 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 633004 T2628 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 633004 T2624 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 633006 T2640 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 633005 T2642 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 633005 T2636 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 633005 T2616 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 633005 T2622 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 633005 T2626 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 633007 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 633009 T2648 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cbdb0f0 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 633009 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 633011 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 633015 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
[junit4] 2> 633020 T2622 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633020 T2616 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633020 T2636 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633020 T2644 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633022 T2638 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633022 T2646 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633020 T2642 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633020 T2634 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633022 T2648 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633022 T2640 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633022 T2630 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633022 T2624 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633022 T2626 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633020 T2628 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633020 T2632 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 633024 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 633026 T2650 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@592e8e3d name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 633026 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 633028 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 633031 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
[junit4] 2> 633036 T2626 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633036 T2630 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633037 T2628 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633036 T2638 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633036 T2640 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633036 T2624 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633036 T2632 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633036 T2636 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633036 T2642 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633036 T2634 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633038 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 633038 T2644 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633037 T2650 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633037 T2646 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633037 T2622 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633037 T2616 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633037 T2648 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 633040 T2652 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43ba79e6 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 633042 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 633044 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 633047 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
[junit4] 2> 633052 T2646 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633052 T2634 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633052 T2626 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633053 T2652 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633053 T2628 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633053 T2624 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633054 T2638 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633054 T2642 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633053 T2630 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633054 T2644 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633055 T2650 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633054 T2616 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633054 T2636 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633054 T2648 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633054 T2622 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633054 T2640 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633054 T2632 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 633056 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 633057 T2654 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@101ea54d name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 633058 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 633060 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 633063 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
[junit4] 2> 633068 T2616 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633068 T2638 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633068 T2646 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633070 T2650 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633070 T2648 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633068 T2642 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633068 T2626 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633071 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 633070 T2640 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633070 T2654 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633070 T2622 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633070 T2628 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633070 T2644 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633069 T2624 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633069 T2632 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633069 T2636 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633069 T2652 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633069 T2634 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633068 T2630 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 633073 T2656 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48d8f6d9 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 633074 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 633076 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 633079 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
[junit4] 2> 633085 T2648 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633086 T2644 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633087 T2654 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633087 T2636 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633085 T2632 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633085 T2622 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633087 T2640 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633087 T2646 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633087 T2638 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633087 T2652 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633087 T2630 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633087 T2634 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633087 T2642 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633087 T2616 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633086 T2624 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633086 T2650 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633086 T2628 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633086 T2626 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633086 T2656 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 633090 T2658 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6444ec9 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 633088 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 633092 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 633094 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 633098 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
[junit4] 2> 633102 T2634 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633103 T2626 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633103 T2636 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633104 T2642 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633104 T2616 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633105 T2652 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633105 T2632 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633105 T2656 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633105 T2630 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633104 T2624 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633104 T2640 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633103 T2622 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633103 T2650 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633102 T2654 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633106 T2646 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633106 T2644 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633106 T2638 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633105 T2628 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633105 T2648 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633105 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 633105 T2658 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 633107 T2660 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54264f62 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 633110 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 633112 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 633115 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
[junit4] 2> 633121 T2648 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633121 T2632 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633121 T2622 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633121 T2650 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633122 T2652 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633122 T2636 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633122 T2630 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633123 T2640 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633122 T2626 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633122 T2624 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633122 T2642 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633124 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 633123 T2660 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633123 T2616 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633123 T2646 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633123 T2658 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633123 T2628 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633123 T2656 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633127 T2662 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c54596d name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 633123 T2644 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633123 T2634 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633123 T2638 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633122 T2654 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 633127 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 633131 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 633135 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
[junit4] 2> 633140 T2654 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633140 T2648 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633141 T2658 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633141 T2616 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633141 T2626 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633142 T2640 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633142 T2650 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633143 T2660 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633140 T2638 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633143 T2662 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633143 T2656 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633144 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 633143 T2646 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633143 T2632 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633143 T2622 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633143 T2624 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633143 T2642 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633142 T2652 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633142 T2628 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633141 T2634 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633141 T2630 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633141 T2636 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633141 T2644 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 633146 T2664 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@554262bc name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 633149 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 633150 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 633154 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
[junit4] 2> 633159 T2648 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633159 T2658 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633160 T2644 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633160 T2656 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633161 T2650 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633161 T2628 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633160 T2624 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633161 T2626 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633162 T2642 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633162 T2664 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633162 T2630 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633162 T2652 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633161 T2636 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633161 T2640 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633161 T2634 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633161 T2622 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633161 T2662 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633161 T2646 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633161 T2632 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633165 T2666 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fa19943 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 633161 T2654 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633160 T2638 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633160 T2616 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633160 T2660 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 633163 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 633167 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 633169 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 633172 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
[junit4] 2> 633180 T2646 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633180 T2662 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633180 T2628 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633181 T2656 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633181 T2640 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633182 T2654 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633182 T2636 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633180 T2652 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633183 T2554 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 633183 T2634 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633182 T2664 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633182 T2648 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633182 T2632 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633182 T2666 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633182 T2622 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633182 T2626 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633181 T2650 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633181 T2638 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633181 T2658 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633181 T2642 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633181 T2644 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633181 T2624 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633181 T2630 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633181 T2660 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633185 T2668 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d034370 name:ZooKeeperConnection Watcher:127.0.0.1:41353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 633183 T2616 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 633189 T2554 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 633191 T2554 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 633195 T2554 oascc.SolrZkClient.makePath makePath: /live_nodes/node23
[junit4] 2> 633201 T2622 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 633201 T2628 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 633201 T2658 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 633202 T2644 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 633202 T2640 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 633202 T2656 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 633203 T2616 oascc.ZkStateReader$3.process Updating live nodes... (24)
[ju
[...truncated too long message...]
kStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 648699 T2993 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 648705 T2554 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:36740 36740
[junit4] 2> 648755 T2554 oas.SolrTestCaseJ4.tearDown ###Ending testShardLeaderChange
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 648756 T2554 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 648757 T2554 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 649497 T2983 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> 649497 T2983 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> 649965 T2991 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> 649965 T2991 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> 651757 T2554 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=th_TH_TH_#u-nu-thai, timezone=America/Guatemala
[junit4] 2> NOTE: Linux 3.8.0-36-generic amd64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=8,threads=1,free=217187448,total=378011648
[junit4] 2> NOTE: All tests run in this JVM: [TestCoreDiscovery, DeleteInactiveReplicaTest, ConnectionManagerTest, RAMDirectoryFactoryTest, CursorMarkTest, BadIndexSchemaTest, IndexSchemaRuntimeFieldTest, TestArbitraryIndexDir, TestSweetSpotSimilarityFactory, LeaderElectionIntegrationTest, TestSolrDeletionPolicy2, TestCollationFieldDocValues, TestSolrXml, DistributedQueryComponentCustomSortTest, ZkSolrClientTest, DocumentAnalysisRequestHandlerTest, DistributedSuggestComponentTest, BasicZkTest, TestFastOutputStream, TestFileDictionaryLookup, StatsComponentTest, OpenCloseCoreStressTest, TestCollapseQParserPlugin, SimplePostToolTest, SignatureUpdateProcessorFactoryTest, PathHierarchyTokenizerFactoryTest, OutputWriterTest, HdfsLockFactoryTest, TestQuerySenderListener, TestInfoStreamLogging, XsltUpdateRequestHandlerTest, TestBinaryResponseWriter, PrimUtilsTest, TestWordDelimiterFilterFactory, TestDistributedGrouping, TestPhraseSuggestions, TestDistributedSearch, DocValuesMultiTest, OpenExchangeRatesOrgProviderTest, TestFieldTypeResource, TestDFRSimilarityFactory, FieldAnalysisRequestHandlerTest, DirectSolrConnectionTest, CoreContainerCoreInitFailuresTest, PingRequestHandlerTest, MinimalSchemaTest, BadCopyFieldTest, TestOmitPositions, HighlighterTest, TestNumberUtils, TestManagedSchemaFieldResource, TestRandomMergePolicy, TestFiltering, TestShardHandlerFactory, TestRandomFaceting, StandardRequestHandlerTest, TestZkChroot, TestSchemaResource, PluginInfoTest, TestFastWriter, TestFuzzyAnalyzedSuggestions, TimeZoneUtilsTest, NoCacheHeaderTest, TestCodecSupport, TestHashPartitioner, TestFreeTextSuggestions, TestRecovery, XmlUpdateRequestHandlerTest, TestIndexSearcher, TestSolrJ, HdfsRecoveryZkTest, TestRecoveryHdfs, CollectionsAPIDistributedZkTest, DateFieldTest, TestCollationKeyRangeQueries, NotRequiredUniqueKeyTest, TestMultiCoreConfBootstrap, UniqFieldsUpdateProcessorFactoryTest, TestAtomicUpdateErrorCases, TestPartialUpdateDeduplication, TestDefaultSimilarityFactory, DebugComponentTest, TestDynamicFieldResource, TestCopyFieldCollectionResource, DefaultValueUpdateProcessorTest, TestCSVResponseWriter, UnloadDistributedZkTest, OverseerTest]
[junit4] Completed on J1 in 23.40s, 8 tests, 1 error <<< FAILURES!
[...truncated 548 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:451: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1278: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:911: There were test failures: 363 suites, 1570 tests, 1 error, 36 ignored (8 assumptions)
Total time: 44 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseParallelGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure