You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/01 19:36:57 UTC

[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.6.0_45) - Build # 9218 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9218/
Java: 32bit/jdk1.6.0_45 -client -XX:+UseSerialGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestBadConfig

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([22ECAA45961C350B]:0)


REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1

Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at __randomizedtesting.SeedInfo.seed([22ECAA45961C350B:26E425B684B9DA2A]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:428)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:425)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:662)


REGRESSION:  org.apache.solr.core.TestBadConfig.testMissingScriptFile

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([22ECAA45961C350B]:0)




Build Log:
[...truncated 9919 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> 694074 T2904 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-OverseerTest-1391272221984
   [junit4]   2> 694074 T2904 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 694077 T2904 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
   [junit4]   2> 694078 T2904 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 694078 T2905 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 694178 T2904 oasc.ZkTestServer.run start zk server on port:40681
   [junit4]   2> 694179 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 694438 T2911 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a994 name:ZooKeeperConnection Watcher:127.0.0.1:40681/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 694439 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 694440 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 694442 T2913 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d7acd7 name:ZooKeeperConnection Watcher:127.0.0.1:40681 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 694442 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 694445 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 694447 T2915 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c3496e name:ZooKeeperConnection Watcher:127.0.0.1:40681 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 694447 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 694448 T2904 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 694452 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 694456 T2904 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 694459 T2904 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 694462 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 694467 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 694470 T2917 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c753e name:ZooKeeperConnection Watcher:127.0.0.1:40681/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 694471 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 694473 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 694477 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 694481 T2911 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 694481 T2917 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 694483 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 694484 T2919 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@344a4d name:ZooKeeperConnection Watcher:127.0.0.1:40681/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 694484 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 694486 T2904 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 694490 T2904 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 694496 T2904 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:40681_solr
   [junit4]   2> 694496 T2904 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 694500 T2904 oasc.Overseer.start Overseer (id=91178416340336644-127.0.0.1:40681_solr-n_0000000000) starting
   [junit4]   2> 694503 T2904 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 694508 T2904 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 694513 T2904 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 694516 T2921 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 694518 T2921 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 694519 T2920 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 694527 T2920 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 694528 T2920 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 694528 T2920 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 694528 T2920 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 694532 T2919 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 694533 T2917 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 694533 T2911 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 695021 T2904 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 695034 T2904 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 695042 T2919 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 695044 T2904 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 695047 T2920 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 695049 T2911 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 695050 T2917 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 695053 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695054 T2923 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14e601c name:ZooKeeperConnection Watcher:127.0.0.1:40681/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695055 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695055 T2919 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 695058 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695062 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 695067 T2911 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 695068 T2923 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 695072 T2919 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 695073 T2904 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 695075 T2920 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 695076 T2920 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 695082 T2919 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 695087 T2904 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 695096 T2919 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 695111 T2919 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 695216 T2923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 695216 T2911 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 695221 T2904 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 695227 T2911 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 695227 T2923 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 695229 T2904 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40681 40681
   [junit4]   2> 695618 T2904 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 695623 T2904 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
   [junit4]   2> 695623 T2904 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 695624 T2924 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 695724 T2904 oasc.ZkTestServer.run start zk server on port:58372
   [junit4]   2> 695725 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695729 T2930 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@90ab46 name:ZooKeeperConnection Watcher:127.0.0.1:58372 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695730 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695761 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695762 T2932 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@101abbc name:ZooKeeperConnection Watcher:127.0.0.1:58372 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695762 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695762 T2904 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 695766 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695768 T2934 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be245 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695768 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695769 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 695773 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695774 T2936 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19a052b name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695774 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695775 T2904 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 695779 T2904 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 695784 T2904 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:58372_solr
   [junit4]   2> 695785 T2904 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 695789 T2904 oasc.Overseer.start Overseer (id=91178416441720835-127.0.0.1:58372_solr-n_0000000000) starting
   [junit4]   2> 695792 T2904 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 695799 T2904 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 695805 T2904 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 695809 T2938 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 695810 T2904 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 695811 T2938 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 695812 T2937 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 695816 T2904 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 695818 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695822 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695823 T2940 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b3eca2 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695824 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695825 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695829 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
   [junit4]   2> 695833 T2934 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 695833 T2940 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 695834 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695835 T2942 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cb37f6 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695836 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695837 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695841 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 695846 T2934 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 695847 T2940 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 695847 T2942 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 695848 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695849 T2944 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11864ed name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695849 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695851 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695854 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 695859 T2934 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 695859 T2944 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 695860 T2942 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 695860 T2940 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 695861 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695862 T2946 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1291c76 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695863 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695864 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695867 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
   [junit4]   2> 695872 T2944 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 695872 T2940 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 695873 T2934 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 695874 T2942 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 695874 T2946 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 695876 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695876 T2948 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e804c6 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695877 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695878 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695881 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
   [junit4]   2> 695888 T2942 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 695888 T2944 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 695888 T2940 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 695889 T2948 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 695890 T2946 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 695890 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695890 T2934 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 695892 T2950 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b4691d name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695892 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695894 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695897 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
   [junit4]   2> 695904 T2948 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 695904 T2944 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 695904 T2946 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 695904 T2934 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 695905 T2940 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 695906 T2942 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 695906 T2950 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 695907 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695908 T2952 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dccaed name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695909 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695910 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695914 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
   [junit4]   2> 695921 T2944 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 695922 T2934 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 695921 T2948 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 695922 T2950 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 695922 T2940 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 695922 T2952 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 695924 T2942 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 695924 T2946 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 695924 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695926 T2954 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f7bc15 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695926 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695927 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695931 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
   [junit4]   2> 695937 T2934 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 695939 T2940 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 695939 T2944 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 695939 T2950 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 695939 T2952 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 695940 T2954 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 695939 T2948 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 695941 T2942 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 695942 T2946 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 695942 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695943 T2956 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18a1317 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695943 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695945 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695948 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
   [junit4]   2> 695956 T2950 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 695957 T2952 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 695957 T2946 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 695957 T2954 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 695957 T2948 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 695958 T2956 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 695958 T2944 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 695958 T2934 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 695959 T2942 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 695959 T2940 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 695960 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695961 T2958 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54827e name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695962 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695963 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695967 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
   [junit4]   2> 695975 T2942 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 695976 T2948 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 695976 T2940 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 695976 T2950 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 695976 T2954 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 695976 T2946 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 695977 T2952 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 695977 T2944 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 695977 T2934 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 695978 T2956 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 695979 T2958 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 695980 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695981 T2960 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@194124d name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695981 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695982 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695986 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
   [junit4]   2> 695997 T2956 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 695997 T2950 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 695998 T2958 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 695997 T2934 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 695997 T2952 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 695998 T2946 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 695998 T2954 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 695999 T2944 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 695999 T2948 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 695999 T2940 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 696000 T2942 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 696000 T2960 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 696001 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696003 T2962 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ce861 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696003 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696005 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696009 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
   [junit4]   2> 696019 T2944 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 696020 T2950 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 696020 T2948 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 696020 T2934 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 696021 T2940 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 696021 T2962 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 696021 T2952 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 696021 T2960 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 696022 T2958 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 696022 T2942 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 696022 T2946 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 696024 T2954 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 696024 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696024 T2956 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 696026 T2964 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@138d328 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696026 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696028 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696031 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
   [junit4]   2> 696039 T2940 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 696055 T2944 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 696056 T2952 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 696057 T2958 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 696058 T2956 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 696058 T2948 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 696058 T2960 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 696059 T2934 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 696059 T2962 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 696060 T2946 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 696060 T2964 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 696061 T2954 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 696061 T2950 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 696061 T2942 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 696062 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696063 T2966 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b1f51c name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696064 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696066 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696069 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
   [junit4]   2> 696080 T2960 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696082 T2964 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696082 T2940 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696083 T2950 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696083 T2962 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696083 T2944 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696084 T2958 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696084 T2952 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696084 T2946 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696085 T2934 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696084 T2942 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696084 T2956 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696085 T2966 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696087 T2954 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696087 T2948 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 696088 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696089 T2968 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cc7ce name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696089 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696091 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696095 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
   [junit4]   2> 696107 T2950 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696108 T2940 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696109 T2942 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696109 T2968 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696109 T2962 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696109 T2960 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696110 T2956 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696109 T2966 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696109 T2944 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696110 T2934 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696110 T2946 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696110 T2952 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696110 T2958 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696111 T2948 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696112 T2964 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696113 T2954 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 696113 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696114 T2970 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ffccbe name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696115 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696116 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696120 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
   [junit4]   2> 696132 T2964 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696133 T2952 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696134 T2948 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696134 T2962 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696135 T2946 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696135 T2956 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696135 T2934 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696135 T2942 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696136 T2950 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696135 T2940 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696136 T2968 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696137 T2970 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696137 T2944 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696137 T2966 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696137 T2960 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696139 T2954 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696139 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696139 T2958 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 696140 T2972 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1edb436 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696141 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696142 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696146 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
   [junit4]   2> 696158 T2960 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696160 T2954 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696160 T2942 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696161 T2958 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696161 T2934 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696161 T2940 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696161 T2950 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696162 T2972 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696162 T2944 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696162 T2962 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696162 T2966 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696163 T2970 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696163 T2952 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696163 T2946 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696163 T2968 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696163 T2948 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696165 T2964 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696165 T2956 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 696166 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696167 T2974 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@539bf8 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696168 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696170 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696173 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
   [junit4]   2> 696187 T2964 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696188 T2940 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696189 T2934 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696189 T2954 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696189 T2944 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696189 T2960 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696189 T2946 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696189 T2962 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696190 T2970 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696190 T2966 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696190 T2948 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696191 T2958 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696190 T2952 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696191 T2942 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696191 T2956 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696191 T2950 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696192 T2974 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696193 T2968 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696194 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696194 T2972 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 696195 T2976 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79d4a7 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696196 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696198 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696203 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
   [junit4]   2> 696218 T2964 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696219 T2950 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696219 T2948 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696219 T2954 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696219 T2944 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696221 T2968 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696221 T2934 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696222 T2956 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696222 T2962 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696222 T2946 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696222 T2942 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696223 T2972 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696224 T2970 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696224 T2960 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696224 T2976 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696224 T2966 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696225 T2958 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696224 T2952 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696226 T2940 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696227 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696227 T2974 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 696228 T2978 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1597ce8 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696228 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696230 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696234 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
   [junit4]   2> 696247 T2964 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696248 T2968 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696248 T2950 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696249 T2954 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696249 T2974 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696250 T2966 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696250 T2934 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696251 T2956 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696251 T2948 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696251 T2942 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696251 T2972 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696252 T2944 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696252 T2970 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696252 T2978 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696252 T2962 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696253 T2952 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696252 T2960 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696253 T2976 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696253 T2958 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696255 T2946 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696255 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696255 T2940 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 696258 T2980 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1592831 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696258 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696260 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696264 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
   [junit4]   2> 696279 T2954 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696281 T2940 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696281 T2974 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696282 T2964 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696282 T2950 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696282 T2962 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696282 T2976 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696283 T2952 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696283 T2958 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696283 T2978 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696283 T2956 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696284 T2942 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696283 T2934 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696284 T2980 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696284 T2972 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696284 T2948 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696284 T2946 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696285 T2970 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696286 T2944 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696286 T2960 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696287 T2968 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696288 T2966 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 696288 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696289 T2982 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1972a76 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696289 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696291 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696296 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
   [junit4]   2> 696311 T2964 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696311 T2948 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696312 T2952 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696313 T2980 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696314 T2968 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696314 T2962 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696314 T2972 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696315 T2966 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696315 T2958 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696315 T2974 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696315 T2946 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696316 T2956 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696315 T2978 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696316 T2934 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696317 T2982 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696317 T2976 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696317 T2960 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696317 T2942 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696318 T2950 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696318 T2944 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696318 T2954 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696320 T2940 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696320 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696320 T2970 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 696322 T2984 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c93e4a name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696322 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696324 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696328 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
   [junit4]   2> 696344 T2972 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696345 T2964 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696345 T2942 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696346 T2934 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696346 T2950 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696347 T2958 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696347 T2968 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696347 T2954 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696348 T2970 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696348 T2952 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696348 T2960 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696348 T2966 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696348 T2944 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696349 T2976 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696349 T2984 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696349 T2982 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696349 T2974 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696350 T2980 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696350 T2948 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696349 T2956 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696349 T2946 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696349 T2978 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696353 T2962 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696353 T2940 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 696354 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696355 T2986 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f42c7 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696356 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696357 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696361 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node23
   [junit4]   2> 696376 T2954 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696381 T2950 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696382 T2952 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696382 T2980 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696383 T2984 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696383 T2978 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696383 T2948 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696384 T2966 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696384 T2942 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696384 T2970 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696385 T2976 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696385 T2958 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696385 T2974 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696385 T2946 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696385 T2956 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696386 T2968 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696386 T2944 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696386 T2940 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696386 T2960 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696386 T2962 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696386 T2964 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696386 T2934 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696388 T2986 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696389 T2972 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696390 T2982 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 696390 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696391 T2988 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fe98a8 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696391 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696393 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696397 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node24
   [junit4]   2> 696416 T2948 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696417 T2950 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696417 T2946 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696418 T2962 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696418 T2934 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696419 T2956 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696419 T2988 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696420 T2970 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696420 T2958 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696420 T2986 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696420 T2966 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696420 T2976 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696421 T2980 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696421 T2984 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696420 T2942 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696420 T2974 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696420 T2940 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696422 T2982 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696422 T2964 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696421 T2968 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696421 T2954 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696421 T2978 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696421 T2944 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696421 T2952 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696424 T2960 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696425 T2972 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 696425 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696426 T2990 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d5d532 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696426 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696428 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696432 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node25
   [junit4]   2> 696452 T2988 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696452 T2934 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696453 T2952 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696453 T2944 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696454 T2972 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696455 T2984 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696456 T2976 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696456 T2966 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696456 T2986 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696456 T2942 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696456 T2970 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696457 T2940 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696457 T2956 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696456 T2974 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696458 T2964 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696456 T2958 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696458 T2962 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696458 T2946 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696458 T2982 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696457 T2954 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696457 T2968 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696457 T2960 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696457 T2980 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696458 T2978 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696461 T2990 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696461 T2950 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696462 T2948 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 696463 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696464 T2992 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b13141 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696464 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696467 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696470 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node26
   [junit4]   2> 696487 T2960 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696488 T2950 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696491 T2988 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696492 T2934 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696492 T2948 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696493 T2964 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696494 T2972 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696494 T2970 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696494 T2966 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696495 T2986 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696495 T2984 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696495 T2974 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696495 T2940 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696495 T2976 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696496 T2980 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696496 T2956 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696495 T2942 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696495 T2958 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696497 T2982 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696497 T2952 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696497 T2962 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696496 T2944 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696496 T2968 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696496 T2954 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696498 T2992 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696497 T2946 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696500 T2990 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696501 T2978 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 696501 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696502 T2994 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c3cd28 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696502 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696504 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696508 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node27
   [junit4]   2> 696524 T2948 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696524 T2966 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696527 T2988 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696527 T2946 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696528 T2978 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696528 T2984 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696528 T2934 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696529 T2972 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696529 T2990 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696529 T2986 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696529 T2970 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696530 T2958 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696530 T2976 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696530 T2940 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696530 T2974 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696531 T2954 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696530 T2980 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696530 T2956 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696530 T2942 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696531 T2960 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696531 T2944 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696531 T2964 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696531 T2992 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696531 T2968 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696532 T2982 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696532 T2962 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696534 T2994 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696534 T2950 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696535 T2952 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 696535 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696536 T2996 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fe7d72 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696537 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696538 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696542 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node28
   [junit4]   2> 696561 T2990 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696562 T2978 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696563 T2948 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696563 T2950 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696563 T2972 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696564 T2996 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696564 T2966 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696564 T2970 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696565 T2976 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696565 T2958 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696565 T2980 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696565 T2956 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696565 T2994 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696566 T2984 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696566 T2968 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696566 T2954 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696567 T2964 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696566 T2940 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696566 T2942 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696565 T2974 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696565 T2986 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696567 T2952 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696567 T2982 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696567 T2962 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696567 T2960 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696567 T2944 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696566 T2934 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696566 T2992 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696571 T2988 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696572 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696571 T2946 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 696573 T2998 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a5070d name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696573 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696575 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696579 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node29
   [junit4]   2> 696610 T2960 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696610 T2988 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696610 T2944 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696611 T2978 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696612 T2992 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696612 T2986 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696612 T2948 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696613 T2972 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696613 T2934 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696613 T2950 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696613 T2970 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696614 T2976 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696614 T2958 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696614 T2966 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696614 T2980 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696615 T2954 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696614 T2942 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696614 T2956 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696614 T2974 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696616 T2982 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696615 T2962 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696615 T2964 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696615 T2994 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696615 T2968 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696615 T2984 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696614 T2940 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696616 T2998 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696616 T2996 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696616 T2952 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696619 T2990 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696619 T2946 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 696619 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696620 T3000 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@100e6f1 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696621 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696622 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696626 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node30
   [junit4]   2> 696641 T2986 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696644 T2988 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696645 T2972 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696645 T2950 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696646 T2996 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696646 T2966 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696647 T2990 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696648 T2980 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696648 T2984 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696648 T2940 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696648 T2954 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696648 T2994 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696648 T2992 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696649 T2998 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696649 T2944 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696649 T2964 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696649 T2978 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696650 T2946 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696649 T2982 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696649 T2952 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696649 T2960 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696649 T2962 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696650 T2948 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696651 T3000 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696652 T2970 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696652 T2976 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696652 T2974 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696652 T2942 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696652 T2958 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696652 T2956 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696654 T2934 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696654 T2968 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 696654 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696656 T3002 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a99a86 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696656 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696657 T2904 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696661 T2904 oascc.SolrZkClient.makePath makePath: /live_nodes/node31
   [junit4]   2> 696679 T2988 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 696679 T2972 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 696680 T2950 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 696682 T2992 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 696683 T2968 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 696683 T2994 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 696683 T2990 oascc.ZkStateReader$3.process Updating live nodes... 

[...truncated too long message...]

.initCore(SolrTestCaseJ4.java:280)
   [junit4]   2> 	        at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:267)
   [junit4]   2> 	        at org.apache.solr.core.AbstractBadConfigTestBase.assertConfigs(AbstractBadConfigTestBase.java:54)
   [junit4]   2> 	        at org.apache.solr.core.AbstractBadConfigTestBase.assertConfigs(AbstractBadConfigTestBase.java:36)
   [junit4]   2> 	        at org.apache.solr.core.TestBadConfig.testUpdateLogButNoVersionField(TestBadConfig.java:51)
   [junit4]   2> 	        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   [junit4]   2> 	        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   [junit4]   2> 	        at java.lang.reflect.Method.invoke(Method.java:597)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:662)
   [junit4]   2> NOTE: test params are: codec=Lucene45, sim=DefaultSimilarity, locale=en_IE, timezone=America/North_Dakota/Beulah
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Sun Microsystems Inc. 1.6.0_45 (32-bit)/cpus=8,threads=2,free=61280744,total=201199616
   [junit4]   2> NOTE: All tests run in this JVM: [UUIDFieldTest, DistributedQueryElevationComponentTest, DirectUpdateHandlerOptimizeTest, BinaryUpdateRequestHandlerTest, MoreLikeThisHandlerTest, TestSolrXmlPersistence, MBeansHandlerTest, TestSolrXmlPersistor, TestStressUserVersions, StatelessScriptUpdateProcessorFactoryTest, TestManagedSchema, ResourceLoaderTest, TestLMJelinekMercerSimilarityFactory, TestAnalyzeInfixSuggestions, URLClassifyProcessorTest, TestLMDirichletSimilarityFactory, BJQParserTest, TestPerFieldSimilarity, TestRealTimeGet, TestFaceting, TestCollapseQParserPlugin, TestPseudoReturnFields, TestInfoStreamLogging, TriLevelCompositeIdRoutingTest, SolrXmlInZkTest, SpellPossibilityIteratorTest, UpdateParamsTest, TestDocBasedVersionConstraints, CacheHeaderTest, TestCharFilters, OverseerCollectionProcessorTest, TestFieldTypeCollectionResource, TestFunctionQuery, TestSuggestSpellingConverter, BasicFunctionalityTest, TestSolrQueryParserResource, TestRemoteStreaming, ShowFileRequestHandlerTest, FieldMutatingUpdateProcessorTest, ParsingFieldUpdateProcessorsTest, CSVRequestHandlerTest, SchemaVersionSpecificBehaviorTest, RecoveryZkTest, UpdateRequestProcessorFactoryTest, CoreAdminCreateDiscoverTest, SOLR749Test, TestMergePolicyConfig, SolrIndexSplitterTest, TestValueSourceCache, DistributedSpellCheckComponentTest, InfoHandlerTest, TestSchemaVersionResource, DeleteShardTest, EchoParamsTest, TestConfig, AssignTest, SearchHandlerTest, SuggesterTest, HdfsLockFactoryTest, LukeRequestHandlerTest, UnloadDistributedZkTest, TestDocSet, BasicDistributedZkTest, RegexBoostProcessorTest, PolyFieldTest, TestWriterPerf, TestRandomMergePolicy, TestCopyFieldCollectionResource, TestBM25SimilarityFactory, HdfsBasicDistributedZk2Test, SystemInfoHandlerTest, FastVectorHighlighterTest, PeerSyncTest, WordBreakSolrSpellCheckerTest, SimpleFacetsTest, CircularListTest, TestJoin, ChangedSchemaMergeTest, DocumentBuilderTest, TestMaxScoreQueryParser, IndexSchemaTest, TestSchemaNameResource, TestExtendedDismaxParser, DeleteReplicaTest, TestTrie, LeaderElectionTest, SyncSliceTest, OverseerTest, ShardRoutingTest, BasicZkTest, FullSolrCloudDistribCmdsTest, TestRandomFaceting, ZkCLITest, ShardRoutingCustomTest, TestDistributedSearch, TestRecovery, TestMultiCoreConfBootstrap, TestStressVersions, TestRangeQuery, TestSolr4Spatial, TestGroupingSearch, TestSort, DirectUpdateHandlerTest, TestBadConfig]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBadConfig -Dtests.seed=22ECAA45961C350B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_IE -Dtests.timezone=America/North_Dakota/Beulah -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | TestBadConfig (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([22ECAA45961C350B]:0)
   [junit4] Completed on J0 in 7223.19s, 7 tests, 2 errors <<< FAILURES!

[...truncated 12 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:439: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1308: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:941: There were test failures: 355 suites, 1543 tests, 1 suite-level error, 2 errors, 35 ignored (7 assumptions)

Total time: 162 minutes 30 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.6.0_45 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure