You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/08/07 13:02:40 UTC
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0_11) - Build # 10849
- Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10849/
Java: 32bit/jdk1.8.0_11 -server -XX:+UseConcMarkSweepGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.OverseerTest.testOverseerFailure
Error Message:
KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
Stack Trace:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
at __randomizedtesting.SeedInfo.seed([30A39531EA789C8C:34AB1AC2F8DD73AD]:0)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:285)
at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:282)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:282)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11727 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.OverseerTest-30A39531EA789C8C-001/init-core-data-001
[junit4] 2> 1362877 T3964 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 1362877 T3964 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 1362877 T3964 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 1362879 T3964 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
[junit4] 2> 1362880 T3964 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1362881 T3965 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1362980 T3964 oasc.ZkTestServer.run start zk server on port:50257
[junit4] 2> 1362981 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1362986 T3971 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d4825c name:ZooKeeperConnection Watcher:127.0.0.1:50257 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1362986 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1362988 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1362989 T3973 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1384ef8 name:ZooKeeperConnection Watcher:127.0.0.1:50257 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1362989 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1362990 T3964 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1362993 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1362994 T3975 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11a9859 name:ZooKeeperConnection Watcher:127.0.0.1:50257/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1362994 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1362994 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1362996 T3964 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1362997 T3964 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1362998 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1362999 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1363000 T3977 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@167327c name:ZooKeeperConnection Watcher:127.0.0.1:50257/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1363001 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1363001 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1363002 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1363004 T3979 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 1363004 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1363004 T3978 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 1363006 T3981 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15f93ae name:ZooKeeperConnection Watcher:127.0.0.1:50257/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1363006 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1363007 T3964 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1363008 T3964 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1363009 T3964 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50257_solr
[junit4] 2> 1363010 T3964 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1363010 T3964 oasc.Overseer.start Overseer (id=92235879126859780-127.0.0.1:50257_solr-n_0000000000) starting
[junit4] 2> 1363011 T3964 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1363013 T3964 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1363014 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1363015 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1363016 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1363018 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1363020 T3983 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1363020 T3982 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1364022 T3984 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1364023 T3982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1364023 T3982 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1364024 T3982 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 1364024 T3982 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1364025 T3984 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1364026 T3978 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1364026 T3979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1364522 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1364527 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1364529 T3984 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1364529 T3964 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1364530 T3982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1364531 T3984 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1364531 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1364531 T3982 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1364532 T3982 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1364533 T3984 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1364633 T3978 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1364633 T3979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1365021 T3983 oasc.OverseerCollectionProcessor.run WARN Overseer cannot talk to ZK
[junit4] 2> 1365687 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1365692 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1365693 T3986 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1748f3 name:ZooKeeperConnection Watcher:127.0.0.1:50257/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1365693 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1365694 T3982 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:50257_solr
[junit4] 2> 1365694 T3987 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
[junit4] 2>
[junit4] 2> 1365695 T3964 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50257_solr
[junit4] 2> 1365695 T3964 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1365696 T3964 oasc.Overseer.start Overseer (id=92235879126859781-127.0.0.1:50257_solr-n_0000000001) starting
[junit4] 2> 1365701 T3989 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1365701 T3988 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1365702 T3988 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1365703 T3988 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1365703 T3988 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1365704 T3990 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1365705 T3978 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1365705 T3979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1365706 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1365707 T3988 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1365709 T3964 oasc.LeaderElector.checkIfIamLeader WARN Failed setting watch org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/92235879126859779-node1_core1-n_0000000002
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:148)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
[junit4] 2> at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
[junit4] 2> at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660)
[junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] 2> at java.lang.reflect.Method.invoke(Method.java:483)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4] 2> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1365710 T3979 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1365711 T3991 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:285)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:282)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:282)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:204)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1365712 T3992 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1365713 T3978 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 1365713 T3990 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1365814 T3978 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1367212 T3979 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:144)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:204)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leaders
[junit4] 2> at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:40)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:131)
[junit4] 2> ... 10 more
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leaders
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:234)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:231)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:231)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:442)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:411)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:398)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:136)
[junit4] 2> at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:34)
[junit4] 2> ... 11 more
[junit4] 2>
[junit4] 2> 1367216 T3964 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50257 50257
[junit4] 2> 1367702 T3989 oasc.OverseerCollectionProcessor.run WARN Overseer cannot talk to ZK
[junit4] 2> 1367786 T3965 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50257 50257
[junit4] 2> 1367787 T3964 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
[junit4] 2> 1367788 T3988 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:50257_solr
[junit4] 2> 1367789 T3993 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=30A39531EA789C8C -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_PY -Dtests.timezone=Europe/Amsterdam -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 4.92s J1 | OverseerTest.testOverseerFailure <<<
[junit4] > Throwable #1: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
[junit4] > at __randomizedtesting.SeedInfo.seed([30A39531EA789C8C:34AB1AC2F8DD73AD]:0)
[junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
[junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] > at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:285)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:282)
[junit4] > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:282)
[junit4] > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
[junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
[junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
[junit4] > at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
[junit4] > at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1367800 T3964 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
[junit4] 2> 1367800 T3964 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1367801 T3994 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1367901 T3964 oasc.ZkTestServer.run start zk server on port:43890
[junit4] 2> 1367903 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1367905 T4000 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d1da5 name:ZooKeeperConnection Watcher:127.0.0.1:43890 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1367905 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1367908 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1367909 T4002 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fbe6ec name:ZooKeeperConnection Watcher:127.0.0.1:43890 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1367909 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1367909 T3964 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1367913 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1367914 T4004 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@186cc71 name:ZooKeeperConnection Watcher:127.0.0.1:43890/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1367914 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1367914 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1367916 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1367916 T4006 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10efe64 name:ZooKeeperConnection Watcher:127.0.0.1:43890/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1367917 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1367917 T3964 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1367919 T3964 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1367920 T3964 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:43890_solr
[junit4] 2> 1367921 T3964 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1367922 T3964 oasc.Overseer.start Overseer (id=92235879449427971-127.0.0.1:43890_solr-n_0000000000) starting
[junit4] 2> 1367922 T3964 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1367924 T3964 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1367925 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1367926 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1367927 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1367929 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1367931 T4008 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1367931 T3964 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1367932 T4007 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1367933 T3964 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1367933 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1367935 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1367936 T4010 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14c1355 name:ZooKeeperConnection Watcher:127.0.0.1:43890/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1367936 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1367937 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1367938 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4] 2> 1367940 T4011 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 1367940 T4012 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 1367940 T4013 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1367941 T4007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1367941 T4007 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1367941 T4007 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 1367942 T4007 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1367942 T4013 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1367943 T4011 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1367943 T4012 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1368440 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1368448 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1368453 T4013 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1368454 T4013 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1368454 T4007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1368456 T4007 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1368456 T4007 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1368456 T4007 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1368457 T4013 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1368559 T4011 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1368559 T4012 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1368954 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1368958 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 1368961 T4013 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1368961 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1368961 T4014 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1368962 T4014 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1368962 T4007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1368963 T4007 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1368963 T4007 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1368964 T4007 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1368964 T4014 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1369066 T4011 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1369066 T4012 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1369462 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1369466 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1369469 T4014 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1369469 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1369469 T4013 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1369470 T4013 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1369470 T4007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1369471 T4007 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node4",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1369472 T4007 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1369472 T4007 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1369473 T4013 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1369574 T4011 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1369574 T4012 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1369970 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1369974 T4013 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1369974 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1369974 T4014 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1369975 T4007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1369975 T4007 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core5",
[junit4] 2> "core_node_name":"node5",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1369976 T4007 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1369976 T4007 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1369977 T4014 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1370081 T4012 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1370081 T4011 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1370474 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1370479 T4014 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1370479 T4013 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1370479 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1370479 T4007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1370480 T4007 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core6",
[junit4] 2> "core_node_name":"node6",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1370480 T4007 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1370480 T4007 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1370481 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1370582 T4012 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1370582 T4011 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1370979 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1370984 T4012 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 1370985 T4016 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:285)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:282)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:282)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:204)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1370985 T4017 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:285)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:282)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:282)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:204)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1370985 T4012 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:285)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:282)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:282)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:204)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1370988 T3964 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:43890 43890
[junit4] 2> 1371564 T3994 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:43890 43890
[junit4] 2> 1371565 T3964 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
[junit4] 2> 1371565 T4007 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:43890_solr
[junit4] 2> 1371566 T4018 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
[junit4] 2>
[junit4] 2> 1371569 T3964 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
[junit4] 2> 1371570 T3964 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1371570 T4019 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1371670 T3964 oasc.ZkTestServer.run start zk server on port:54373
[junit4] 2> 1371671 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371673 T4025 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4543d6 name:ZooKeeperConnection Watcher:127.0.0.1:54373 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371673 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371675 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371676 T4027 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c5e0ab name:ZooKeeperConnection Watcher:127.0.0.1:54373 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371676 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371676 T3964 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1371678 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371679 T4029 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10acde9 name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371679 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371680 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1371682 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371682 T4031 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ba8a6 name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371683 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371683 T3964 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1371684 T3964 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1371686 T3964 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:54373_solr
[junit4] 2> 1371686 T3964 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1371687 T3964 oasc.Overseer.start Overseer (id=92235879696367619-127.0.0.1:54373_solr-n_0000000000) starting
[junit4] 2> 1371688 T3964 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1371689 T3964 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1371690 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1371691 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1371693 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1371694 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1371696 T4033 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1371696 T3964 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1371697 T4032 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1371698 T3964 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1371698 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371700 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371701 T4035 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@81ece name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371701 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371702 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371703 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
[junit4] 2> 1371705 T4036 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 1371705 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371705 T4037 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 1371709 T4039 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@907f9c name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371709 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371710 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371712 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1371713 T4036 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1371713 T4037 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1371713 T4040 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1371713 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371714 T4042 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e68da2 name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371714 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371715 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371716 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 1371718 T4037 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1371718 T4043 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1371718 T4036 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1371718 T4040 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1371718 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371722 T4045 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@672ba0 name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371723 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371724 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371725 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
[junit4] 2> 1371726 T4037 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1371726 T4040 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1371727 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371726 T4036 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1371726 T4046 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1371726 T4043 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1371728 T4048 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13becc name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371728 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371729 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371730 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
[junit4] 2> 1371731 T4036 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 1371731 T4037 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 1371731 T4043 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 1371732 T4040 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 1371732 T4049 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 1371732 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371733 T4046 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 1371733 T4051 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16e594d name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371733 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371734 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371735 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
[junit4] 2> 1371737 T4049 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 1371737 T4040 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 1371737 T4052 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 1371737 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371737 T4043 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 1371737 T4037 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 1371737 T4036 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 1371737 T4046 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 1371738 T4054 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5059df name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371738 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371739 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371740 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
[junit4] 2> 1371741 T4036 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1371741 T4046 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1371741 T4052 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1371741 T4043 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1371741 T4037 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1371742 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371742 T4040 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1371742 T4055 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1371742 T4049 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1371743 T4057 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17d485e name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371743 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371744 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371745 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
[junit4] 2> 1371746 T4036 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1371746 T4049 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1371746 T4052 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1371746 T4043 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1371747 T4058 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1371748 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371747 T4040 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1371747 T4055 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1371746 T4037 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1371746 T4046 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1371748 T4060 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a171a3 name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371749 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371749 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371751 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
[junit4] 2> 1371752 T4049 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1371752 T4037 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1371752 T4046 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1371753 T4052 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1371753 T4055 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1371753 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371753 T4040 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1371753 T4063 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c2e10b name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371753 T4043 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1371753 T4058 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1371753 T4036 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1371754 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371753 T4061 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1371755 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371756 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
[junit4] 2> 1371757 T4037 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1371757 T4052 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1371757 T4049 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1371758 T4061 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1371758 T4055 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1371758 T4046 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1371758 T4043 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1371758 T4036 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1371759 T4058 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1371759 T4066 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dec34d name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371759 T4040 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1371758 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371759 T4064 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1371760 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371761 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371762 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
[junit4] 2> 1371763 T4049 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1371763 T4052 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1371763 T4043 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1371764 T4061 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1371764 T4036 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1371764 T4055 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1371764 T4067 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1371764 T4037 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1371764 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371764 T4040 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1371764 T4046 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1371764 T4058 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1371764 T4064 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1371765 T4069 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c49d0 name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371765 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371766 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371767 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
[junit4] 2> 1371769 T4043 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1371769 T4070 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1371769 T4037 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1371770 T4055 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1371769 T4036 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1371769 T4058 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1371770 T4040 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1371770 T4072 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1151ed9 name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371769 T4049 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1371769 T4067 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1371769 T4046 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1371769 T4052 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1371770 T4061 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1371770 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371770 T4064 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1371771 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371772 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371773 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
[junit4] 2> 1371776 T4043 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1371776 T4052 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1371776 T4037 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1371777 T4049 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1371777 T4061 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1371777 T4058 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1371776 T4055 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1371777 T4070 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1371777 T4036 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1371777 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371777 T4064 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1371777 T4046 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1371777 T4040 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1371777 T4067 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1371777 T4073 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1371778 T4075 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1217b09 name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371779 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371780 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371781 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
[junit4] 2> 1371782 T4043 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371782 T4037 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371782 T4040 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371782 T4070 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371784 T4067 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371782 T4064 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371782 T4036 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371782 T4052 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371784 T4078 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371784 T4077 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@781d4b name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371784 T4046 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371783 T4073 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371783 T4049 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371783 T4061 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371783 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371783 T4058 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371783 T4055 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1371786 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371787 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371788 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
[junit4] 2> 1371789 T4043 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371790 T4052 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371790 T4036 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371790 T4064 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371790 T4040 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371791 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371791 T4058 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371791 T4078 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371791 T4049 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371790 T4067 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371790 T4070 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371790 T4061 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371790 T4073 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371790 T4055 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371790 T4079 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371790 T4046 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371790 T4037 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1371791 T4081 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e9c5d name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371793 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371794 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371795 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
[junit4] 2> 1371796 T4036 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371797 T4037 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371797 T4055 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371797 T4061 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371797 T4049 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371797 T4079 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371796 T4067 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371797 T4082 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371797 T4064 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371798 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371797 T4043 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371798 T4040 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371799 T4084 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a6bdd9 name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371797 T4070 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371797 T4046 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371797 T4073 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371797 T4058 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371797 T4052 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371797 T4078 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1371799 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1371800 T3964 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1371802 T3964 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
[junit4] 2> 1371803 T4043 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371803 T4037 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371803 T4061 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371809 T4046 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371809 T4049 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371809 T4067 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371809 T4079 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371809 T3964 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1371809 T4082 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371810 T4052 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371809 T4058 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371809 T4064 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371809 T4085 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371809 T4073 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371809 T4078 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371809 T4036 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371809 T4040 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371809 T4055 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371809 T4070 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1371810 T4087 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1196017 name:ZooKeeperConnection Watcher:127.0.0.1:54373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1371812 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKee
[...truncated too long message...]
on Watcher:127.0.0.1:60941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1434768 T3964 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1434769 T3964 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1434770 T3964 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1434771 T3964 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60941_solr
[junit4] 2> 1434771 T3964 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1434772 T3964 oasc.Overseer.start Overseer (id=92235883830378500-127.0.0.1:60941_solr-n_0000000000) starting
[junit4] 2> 1434773 T3964 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1434775 T3964 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1434776 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1434777 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1434779 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1434780 T3964 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1434782 T4910 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1434783 T4909 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1434784 T4909 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1434784 T4909 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"12",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1434784 T4909 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
[junit4] 2> 1434785 T4909 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1434786 T4911 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1434786 T4906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1434786 T4905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1435283 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1435287 T3964 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1435289 T4911 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1435290 T3964 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1435290 T4909 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1435291 T4905 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 1435292 T4906 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 1435293 T3964 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60941 60941
[junit4] 2> 1436055 T4892 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60941 60941
[junit4] 2> 1436058 T3964 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
[junit4] 2> 1436059 T4909 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:60941_solr
[junit4] 2> 1436059 T4912 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
[junit4] 2>
[junit4] 2> 1439061 T3964 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.OverseerTest-30A39531EA789C8C-001
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=es_PY, timezone=Europe/Amsterdam
[junit4] 2> NOTE: Linux 3.13.0-32-generic i386/Oracle Corporation 1.8.0_11 (32-bit)/cpus=8,threads=1,free=117566712,total=253018112
[junit4] 2> NOTE: All tests run in this JVM: [DeleteReplicaTest, TestChildDocTransformer, LegacyHTMLStripCharFilterTest, TestReversedWildcardFilterFactory, DistributedTermsComponentTest, SortByFunctionTest, TestCollationField, TestReloadAndDeleteDocs, SolrCoreTest, TestCloudInspectUtil, TestStandardQParsers, TestExceedMaxTermLength, CurrencyFieldXmlFileTest, DistributedExpandComponentTest, TestExtendedDismaxParser, TestLFUCache, TestElisionMultitermQuery, TestSchemaSimilarityResource, TestRealTimeGet, URLClassifyProcessorTest, ZkSolrClientTest, TestFileDictionaryLookup, TestFieldTypeResource, TestCoreContainer, TestPHPSerializedResponseWriter, TestUniqueKeyFieldResource, TestInfoStreamLogging, TestLMDirichletSimilarityFactory, TestSolrQueryParserDefaultOperatorResource, TestRandomMergePolicy, SearchHandlerTest, TestUpdate, TestOmitPositions, NumericFieldsTest, TestQueryTypes, DistributedQueryComponentOptimizationTest, DefaultValueUpdateProcessorTest, TimeZoneUtilsTest, TestCSVResponseWriter, MigrateRouteKeyTest, AtomicUpdatesTest, TestDynamicFieldCollectionResource, TestSolrXmlPersistor, DistribDocExpirationUpdateProcessorTest, PolyFieldTest, HdfsWriteToMultipleCollectionsTest, TestGroupingSearch, SchemaVersionSpecificBehaviorTest, HdfsBasicDistributedZkTest, TestFunctionQuery, TestZkChroot, TestNoOpRegenerator, TestCSVLoader, ResponseHeaderTest, TestCollectionAPI, TestWriterPerf, SampleTest, TestLeaderElectionZkExpiry, AnalysisAfterCoreReloadTest, BasicZkTest, SolrIndexConfigTest, TestSimpleQParserPlugin, ShardRoutingCustomTest, OutputWriterTest, DistributedQueryElevationComponentTest, DebugComponentTest, SuggesterFSTTest, TestQueryUtils, ConvertedLegacyTest, DeleteInactiveReplicaTest, TestComplexPhraseQParserPlugin, FieldAnalysisRequestHandlerTest, ChaosMonkeyNothingIsSafeTest, FileBasedSpellCheckerTest, TestIBSimilarityFactory, TestDistributedGrouping, SoftAutoCommitTest, HdfsUnloadDistributedZkTest, TestAtomicUpdateErrorCases, TestFreeTextSuggestions, TestBinaryField, TestRTGBase, TestSystemIdResolver, UpdateRequestProcessorFactoryTest, SystemInfoHandlerTest, BasicDistributedZk2Test, DirectUpdateHandlerOptimizeTest, OpenCloseCoreStressTest, HardAutoCommitTest, SolrPluginUtilsTest, TestRestManager, TestClassNameShortening, DOMUtilTest, ZkNodePropsTest, UniqFieldsUpdateProcessorFactoryTest, UUIDUpdateProcessorFallbackTest, TestSolrXmlPersistence, QueryResultKeyTest, TestFastWriter, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, SyncSliceTest, OverseerTest]
[junit4] Completed on J1 in 76.21s, 9 tests, 1 error <<< FAILURES!
[...truncated 632 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:474: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923: There were test failures: 411 suites, 1711 tests, 1 error, 38 ignored (17 assumptions)
Total time: 89 minutes 51 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_11 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any