You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/11/28 10:40:23 UTC
[JENKINS] Lucene-Solr-4.x-Linux
(32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0) - Build # 8394 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8394/
Java: 32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0 -XnoOpt
1 tests failed.
REGRESSION: org.apache.solr.cloud.OverseerTest.testOverseerFailure
Error Message:
Stack Trace:
org.apache.solr.common.cloud.ZooKeeperException:
at __randomizedtesting.SeedInfo.seed([5E69D4759BCEDEA6:5A615B86896B3187]:0)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:265)
at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:774)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:683)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:44)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:56)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/90808684579127299-node1_core1-n_
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:309)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:231)
at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
... 3 more
Build Log:
[...truncated 9930 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> 1176221 T2878 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1385630560756
[junit4] 2> 1176222 T2878 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 1176246 T2878 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
[junit4] 2> 1176250 T2878 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1176250 T2879 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1176350 T2878 oasc.ZkTestServer.run start zk server on port:48712
[junit4] 2> 1176351 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1176437 T2885 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1add52a5 name:ZooKeeperConnection Watcher:127.0.0.1:48712/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1176438 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1176439 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1176441 T2887 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ae17c6a name:ZooKeeperConnection Watcher:127.0.0.1:48712 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1176441 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1176443 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1176444 T2889 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ae3b2a2 name:ZooKeeperConnection Watcher:127.0.0.1:48712 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1176445 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1176445 T2878 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1176450 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1176454 T2878 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1176457 T2878 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1176460 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1176464 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1176465 T2891 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1adfc900 name:ZooKeeperConnection Watcher:127.0.0.1:48712/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1176466 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1176467 T2878 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1176473 T2878 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1176476 T2878 oasc.Overseer.start Overseer (id=90808684431671299-127.0.0.1:48712_solr-n_0000000000) starting
[junit4] 2> 1176478 T2878 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1176482 T2878 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1176486 T2878 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1176489 T2893 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1176492 T2892 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1176494 T2878 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1176496 T2892 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1176497 T2892 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1176497 T2892 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1176500 T2891 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1176501 T2885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1176547 T2878 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1176549 T2891 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1176551 T2892 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1176552 T2892 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"active"}
[junit4] 2> 1176552 T2892 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
[junit4] 2> 1176553 T2892 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1176556 T2891 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1176659 T2885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1176703 T2878 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48712 48712
[junit4] 2> 1176904 T2878 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1176926 T2878 oas.SolrTestCaseJ4.setUp ###Starting testReplay
[junit4] 2> 1176929 T2878 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1176930 T2894 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1177030 T2878 oasc.ZkTestServer.run start zk server on port:39262
[junit4] 2> 1177031 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1177033 T2900 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b49c103 name:ZooKeeperConnection Watcher:127.0.0.1:39262/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1177033 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1177034 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1177035 T2902 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b4a0927 name:ZooKeeperConnection Watcher:127.0.0.1:39262 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1177035 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1177053 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1177054 T2904 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b4a3675 name:ZooKeeperConnection Watcher:127.0.0.1:39262 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1177055 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1177055 T2878 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1177058 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1177061 T2878 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1177064 T2878 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1177066 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1177071 T2878 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1177075 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1177076 T2906 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b4aef5e name:ZooKeeperConnection Watcher:127.0.0.1:39262/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1177076 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1177077 T2878 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1177082 T2878 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1177085 T2878 oasc.Overseer.start Overseer (id=90808684476235779-127.0.0.1:39262_solr-n_0000000000) starting
[junit4] 2> 1177088 T2878 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1177094 T2878 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1177097 T2908 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1177103 T2907 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1177104 T2907 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
[junit4] 2> 1177104 T2907 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1177105 T2900 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1177111 T2907 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core2",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1177112 T2900 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1177115 T2907 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1177118 T2907 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1177118 T2907 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core3",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1177121 T2906 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1177122 T2900 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1177206 T2878 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39262 39262
[junit4] 2> 1177277 T2878 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1177305 T2878 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
[junit4] 2> 1177307 T2878 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1177307 T2909 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1177407 T2878 oasc.ZkTestServer.run start zk server on port:53106
[junit4] 2> 1177408 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1177410 T2915 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b6c9983 name:ZooKeeperConnection Watcher:127.0.0.1:53106/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1177410 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1177411 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1177412 T2917 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b7f3e75 name:ZooKeeperConnection Watcher:127.0.0.1:53106 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1177412 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1177429 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1177430 T2919 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b7f6be8 name:ZooKeeperConnection Watcher:127.0.0.1:53106 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1177430 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1177431 T2878 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1177434 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1177436 T2878 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1177439 T2878 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1177441 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1177445 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1177446 T2921 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b7fee47 name:ZooKeeperConnection Watcher:127.0.0.1:53106/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1177446 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1177448 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1177451 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1177454 T2915 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 1177455 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1177455 T2921 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 1177456 T2923 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b803d1b name:ZooKeeperConnection Watcher:127.0.0.1:53106/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1177456 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1177457 T2878 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1177464 T2878 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1177466 T2878 oasc.Overseer.start Overseer (id=90808684500942852-127.0.0.1:53106_solr-n_0000000000) starting
[junit4] 2> 1177469 T2878 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1177472 T2878 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1177476 T2878 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1177479 T2925 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1177481 T2924 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1177484 T2923 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1177486 T2924 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1177487 T2924 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1177487 T2924 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 1177487 T2924 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1177490 T2923 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1177491 T2915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1177491 T2921 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1177985 T2878 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1177996 T2878 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1178002 T2923 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1178003 T2878 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1178006 T2924 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1178008 T2915 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 1178009 T2921 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 1178011 T2923 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1178012 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1178013 T2927 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b812da0 name:ZooKeeperConnection Watcher:127.0.0.1:53106/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1178014 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1178015 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1178018 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1178022 T2915 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 1178022 T2927 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 1178024 T2923 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1178024 T2878 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1178026 T2924 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1178026 T2924 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1178033 T2923 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1178036 T2878 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1178042 T2923 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1178049 T2923 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1178153 T2927 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1178153 T2915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1178154 T2878 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1178160 T2915 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 1178160 T2927 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 1178163 T2878 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53106 53106
[junit4] 2> 1178475 T2878 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1178497 T2878 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
[junit4] 2> 1178499 T2878 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1178499 T2928 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1178599 T2878 oasc.ZkTestServer.run start zk server on port:51163
[junit4] 2> 1178600 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1178602 T2934 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b9f414f name:ZooKeeperConnection Watcher:127.0.0.1:51163/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1178602 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1178603 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1178604 T2936 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b9f6992 name:ZooKeeperConnection Watcher:127.0.0.1:51163 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1178604 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1178641 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1178642 T2938 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b9f9bea name:ZooKeeperConnection Watcher:127.0.0.1:51163 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1178642 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1178643 T2878 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1178645 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1178649 T2878 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1178653 T2878 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1178655 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1178658 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1178659 T2940 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ba01e41 name:ZooKeeperConnection Watcher:127.0.0.1:51163/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1178660 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1178661 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1178664 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1178667 T2934 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 1178667 T2940 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 1178668 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1178669 T2942 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ba06d17 name:ZooKeeperConnection Watcher:127.0.0.1:51163/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1178669 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1178670 T2878 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1178675 T2878 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1178678 T2878 oasc.Overseer.start Overseer (id=90808684579127300-127.0.0.1:51163_solr-n_0000000000) starting
[junit4] 2> 1178680 T2878 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1178684 T2878 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1178687 T2878 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1178690 T2944 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1178691 T2943 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1179693 T2942 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1179695 T2943 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1179696 T2943 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1179696 T2943 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 1179696 T2943 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1179700 T2942 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1179701 T2940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1179701 T2934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1180193 T2878 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1180203 T2878 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1180210 T2942 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1180211 T2878 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1180213 T2943 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1180216 T2942 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1180216 T2878 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1180220 T2943 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1180221 T2943 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1180225 T2942 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1180329 T2940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1180329 T2934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1181385 T2878 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1181392 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1181393 T2946 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bc1bbeb name:ZooKeeperConnection Watcher:127.0.0.1:51163/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1181393 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1181396 T2878 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1181399 T2878 oasc.Overseer.start Overseer (id=90808684579127301-127.0.0.1:51163_solr-n_0000000001) starting
[junit4] 2> 1181404 T2948 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1181405 T2947 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1181411 T2947 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1181412 T2947 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1181413 T2947 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1181417 T2946 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1181418 T2940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1181424 T2934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1181478 T2946 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1181478 T2878 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1181481 T2947 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1181490 T2940 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1181497 T2947 oasc.SolrException.log ERROR Problem cleaning up collection in zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /collections/collection1/leaders
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:169)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:166)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:166)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:650)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:645)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:875)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:224)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:185)
[junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4] 2>
[junit4] 2> 1181501 T2946 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1181502 T2940 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:252)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:249)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:249)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> 1181504 T2947 oasc.Overseer$ClusterStateUpdater.setShardLeader ERROR Could not mark shard leader for non existing collection:collection1
[junit4] 2> 1181508 T2946 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1181610 T2940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1181610 T2934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1182569 T2934 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 1182569 T2940 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 1182572 T2878 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51163 51163
[junit4] 2> 1182803 T2878 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=5E69D4759BCEDEA6 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_PH -Dtests.timezone=Singapore -Dtests.file.encoding=US-ASCII
[junit4] ERROR 4.34s J1 | OverseerTest.testOverseerFailure <<<
[junit4] > Throwable #1: org.apache.solr.common.cloud.ZooKeeperException:
[junit4] > at __randomizedtesting.SeedInfo.seed([5E69D4759BCEDEA6:5A615B86896B3187]:0)
[junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:265)
[junit4] > at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
[junit4] > at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
[junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4] > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/90808684579127299-node1_core1-n_
[junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
[junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:309)
[junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:231)
[junit4] > at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
[junit4] > at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
[junit4] > ... 3 more
[junit4] 2> 1182837 T2878 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
[junit4] 2> 1182842 T2878 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1182842 T2949 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1182942 T2878 oasc.ZkTestServer.run start zk server on port:48357
[junit4] 2> 1182943 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1182945 T2955 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b54ff4d name:ZooKeeperConnection Watcher:127.0.0.1:48357 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1182945 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1182963 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1182964 T2957 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b562e38 name:ZooKeeperConnection Watcher:127.0.0.1:48357 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1182964 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1182965 T2878 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1182970 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1182971 T2959 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b5687ba name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1182972 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1182972 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1182974 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1182975 T2961 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b570129 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1182975 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1182976 T2878 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1182982 T2878 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1182985 T2878 oasc.Overseer.start Overseer (id=90808684863815683-127.0.0.1:48357_solr-n_0000000000) starting
[junit4] 2> 1182987 T2878 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1182991 T2878 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1182995 T2878 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1182997 T2963 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1183000 T2878 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1183001 T2962 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1183004 T2878 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1183006 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183009 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183010 T2965 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b5ddd72 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183010 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183012 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183015 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
[junit4] 2> 1183019 T2959 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 1183019 T2965 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 1183020 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183021 T2967 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b5eac84 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183022 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183023 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183026 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1183034 T2965 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1183034 T2959 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1183034 T2967 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1183035 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183036 T2969 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b8a26a2 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183036 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183038 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183041 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 1183045 T2959 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1183045 T2965 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1183045 T2967 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1183046 T2969 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1183047 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183048 T2971 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b8b95c9 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183048 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183049 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183053 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
[junit4] 2> 1183057 T2969 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1183057 T2965 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1183058 T2967 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1183058 T2959 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1183058 T2971 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1183059 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183060 T2973 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b185978 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183060 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183062 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183065 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
[junit4] 2> 1183070 T2965 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 1183070 T2967 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 1183070 T2973 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 1183071 T2969 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 1183071 T2959 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 1183072 T2971 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 1183072 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183073 T2975 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b197522 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183073 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183075 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183079 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
[junit4] 2> 1183112 T2973 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 1183117 T2965 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 1183118 T2975 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 1183119 T2969 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 1183119 T2959 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 1183120 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183121 T2977 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bd67236 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183121 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183122 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183124 T2967 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 1183124 T2971 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 1183126 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
[junit4] 2> 1183131 T2975 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1183132 T2959 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1183134 T2971 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1183134 T2967 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1183134 T2965 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1183135 T2969 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1183135 T2973 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1183135 T2977 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 1183136 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183137 T2979 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bdddcd9 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183138 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183139 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183142 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
[junit4] 2> 1183149 T2979 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1183149 T2977 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1183149 T2967 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1183150 T2971 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1183150 T2965 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1183150 T2973 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1183150 T2959 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1183151 T2969 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1183151 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183152 T2975 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 1183153 T2981 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1af78893 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183153 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183154 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183158 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
[junit4] 2> 1183164 T2975 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1183164 T2977 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1183164 T2971 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1183165 T2967 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1183165 T2979 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1183166 T2959 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1183167 T2973 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1183167 T2969 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1183167 T2965 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1183167 T2981 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 1183168 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183169 T2983 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be415f3 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183170 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183171 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183175 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
[junit4] 2> 1183185 T2975 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1183185 T2959 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1183186 T2969 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1183186 T2967 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1183187 T2973 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1183187 T2981 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1183187 T2977 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1183188 T2971 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1183188 T2965 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1183189 T2979 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1183190 T2983 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 1183190 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183192 T2985 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be46574 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183193 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183194 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183197 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
[junit4] 2> 1183204 T2971 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1183205 T2975 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1183205 T2983 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1183206 T2977 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1183205 T2979 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1183206 T2969 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1183206 T2967 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1183207 T2973 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1183207 T2981 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1183208 T2959 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1183208 T2965 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1183210 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183210 T2985 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 1183211 T2987 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be4b509 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183211 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183212 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183215 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
[junit4] 2> 1183225 T2977 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1183227 T2959 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1183228 T2965 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1183229 T2979 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1183229 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183229 T2973 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1183229 T2969 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1183231 T2981 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1183230 T2975 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1183230 T2987 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1183231 T2989 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be50482 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183232 T2985 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1183232 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183234 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183234 T2967 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1183236 T2983 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1183237 T2971 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 1183238 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
[junit4] 2> 1183241 T2981 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1183243 T2969 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1183248 T2973 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1183249 T2959 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1183253 T2967 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1183255 T2987 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1183256 T2971 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1183256 T2975 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1183257 T2985 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1183257 T2977 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1183258 T2979 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1183257 T2983 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1183257 T2965 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1183258 T2989 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 1183259 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183260 T2991 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be55433 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183260 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183262 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183265 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
[junit4] 2> 1183270 T2983 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183271 T2973 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183272 T2971 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183272 T2959 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183273 T2969 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183274 T2987 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183274 T2979 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183274 T2977 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183275 T2965 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183275 T2981 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183276 T2975 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183277 T2967 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183277 T2991 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183278 T2985 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183278 T2989 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 1183279 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183280 T2993 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be5a3c6 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183280 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183282 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183286 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
[junit4] 2> 1183298 T2971 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183300 T2977 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183302 T2967 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183302 T2993 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183303 T2985 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183305 T2973 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183305 T2981 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183305 T2983 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183306 T2989 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183306 T2975 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183305 T2987 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183308 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183308 T2959 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183308 T2995 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be5f385 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183308 T2991 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183309 T2969 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183309 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183315 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183317 T2965 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183319 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
[junit4] 2> 1183322 T2979 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 1183324 T2967 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183326 T2975 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183334 T2983 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183334 T2993 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183335 T2981 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183336 T2965 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183336 T2971 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183336 T2969 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183336 T2979 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183337 T2987 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183337 T2977 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183337 T2989 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183337 T2995 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183338 T2985 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183338 T2973 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183338 T2991 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183338 T2959 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 1183339 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183340 T2997 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be64338 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183340 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183342 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183345 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
[junit4] 2> 1183354 T2993 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183355 T2983 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183356 T2991 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183356 T2977 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183357 T2975 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183358 T2979 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183359 T2981 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183359 T2989 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183359 T2967 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183360 T2969 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183361 T2987 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183361 T2959 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183361 T2971 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183361 T2995 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183363 T2985 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183363 T2973 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183364 T2965 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183364 T2997 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 1183364 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183365 T2999 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be692e2 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183365 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183367 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183370 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
[junit4] 2> 1183376 T2977 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183377 T2995 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183377 T2979 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183378 T2987 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183378 T2991 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183379 T2983 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183381 T2965 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183381 T2973 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183381 T2959 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183382 T2981 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183382 T2967 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183383 T2969 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183383 T2993 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183384 T2975 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183384 T2997 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183384 T2989 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183385 T2985 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183385 T2999 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183385 T2971 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 1183386 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183387 T3001 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bdf987b name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183387 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183389 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183392 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
[junit4] 2> 1183396 T2965 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183397 T2991 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183398 T2981 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183399 T2977 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183399 T2973 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183400 T2989 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183401 T2997 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183401 T2975 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183401 T2983 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183402 T2979 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183403 T2995 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183403 T2987 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183403 T3001 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183404 T2999 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183406 T2959 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183406 T2993 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183406 T2969 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183407 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183407 T2967 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183407 T2985 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183408 T3003 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c192dd8 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183408 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183408 T2971 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 1183409 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183413 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
[junit4] 2> 1183418 T2995 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183419 T2971 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183419 T3001 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183420 T2981 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183421 T2993 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183421 T2997 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183422 T2975 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183422 T2965 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183423 T2989 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183423 T2991 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183424 T2973 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183426 T2983 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183426 T2985 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183426 T2959 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183427 T2999 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183427 T2979 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183427 T2987 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183428 T2969 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183429 T3003 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183429 T2967 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183430 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183431 T3005 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c197d84 name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183432 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183432 T2977 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 1183433 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183436 T2878 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
[junit4] 2> 1183456 T2991 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183460 T2973 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183462 T2969 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183463 T2999 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183463 T3005 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183464 T2997 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183464 T2959 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183464 T2985 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183464 T2983 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183465 T2993 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183465 T2987 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183465 T2967 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183466 T2971 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183466 T2878 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1183467 T3007 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c19cd3a name:ZooKeeperConnection Watcher:127.0.0.1:48357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1183467 T2878 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1183468 T2965 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183468 T2989 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183469 T2878 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1183471 T2981 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183473 T2995 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183473 T2979 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183473 T3001 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 1183473 T2977 oascc.ZkStateReader$3.proces
[...truncated too long message...]
va:249)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:249)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> 1208259 T3379 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:252)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:249)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:249)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> 1208260 T3379 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:252)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:249)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:249)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> 1208745 T3387 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:252)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:249)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:249)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> 1208746 T3387 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:252)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:249)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:249)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> 1210743 T2878 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=en_PH, timezone=Singapore
[junit4] 2> NOTE: Linux 3.8.0-33-generic i386/Oracle Corporation 1.6.0_45 (32-bit)/cpus=8,threads=1,free=165270256,total=368656384
[junit4] 2> NOTE: All tests run in this JVM: [TestCollationKeyRangeQueries, InfoHandlerTest, TestWordDelimiterFilterFactory, TermVectorComponentDistributedTest, IndexSchemaRuntimeFieldTest, TestDefaultSearchFieldResource, TestDefaultSimilarityFactory, FullSolrCloudDistribCmdsTest, DeleteInactiveReplicaTest, TestCodecSupport, TestCollationField, TestCoreDiscovery, TestPhraseSuggestions, TestBinaryField, TestQuerySenderNoQuery, ConvertedLegacyTest, SolrCoreCheckLockOnStartupTest, ChangedSchemaMergeTest, DistanceFunctionTest, TestHighFrequencyDictionaryFactory, BasicDistributedZk2Test, ChaosMonkeySafeLeaderTest, SolrIndexConfigTest, TestXIncludeConfig, TestAnalyzeInfixSuggestions, TestCloudManagedSchemaAddField, CoreMergeIndexesAdminHandlerTest, HdfsDirectoryTest, DefaultValueUpdateProcessorTest, TestDistribDocBasedVersion, ChaosMonkeyNothingIsSafeTest, HdfsChaosMonkeySafeLeaderTest, TestJmxIntegration, OpenExchangeRatesOrgProviderTest, TestRTGBase, HighlighterConfigTest, TestSchemaNameResource, SolrCmdDistributorTest, CurrencyFieldOpenExchangeTest, TestIBSimilarityFactory, TestMultiCoreConfBootstrap, ModifyConfFileTest, DirectSolrSpellCheckerTest, PreAnalyzedFieldTest, TestFieldResource, EnumFieldTest, TestDistributedSearch, MultiTermTest, UniqFieldsUpdateProcessorFactoryTest, ZkSolrClientTest, TestFastOutputStream, TestAtomicUpdateErrorCases, TestUniqueKeyFieldResource, TestNRTOpen, TestStressRecovery, TestNumberUtils, TestCSVResponseWriter, BasicZkTest, SolrRequestParserTest, TestUtils, TestSolrQueryParserDefaultOperatorResource, SuggesterWFSTTest, NotRequiredUniqueKeyTest, RequiredFieldsTest, NumericFieldsTest, DateFieldTest, SimplePostToolTest, PluginInfoTest, StatsComponentTest, FileUtilsTest, TestInfoStreamLogging, TestStressLucene, TestSolrJ, BJQParserTest, StandardRequestHandlerTest, StressHdfsTest, TestStressUserVersions, ZkNodePropsTest, PathHierarchyTokenizerFactoryTest, SolrPluginUtilsTest, FieldAnalysisRequestHandlerTest, CurrencyFieldXmlFileTest, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, OverseerTest]
[junit4] Completed on J1 in 34.61s, 8 tests, 1 error <<< FAILURES!
[...truncated 534 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:426: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:406: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1279: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:912: There were test failures: 346 suites, 1513 tests, 1 error, 34 ignored (6 assumptions)
Total time: 86 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0 -XnoOpt
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure