You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/07/14 02:16:04 UTC

[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 584 - Failure

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/584/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed

Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
	at __randomizedtesting.SeedInfo.seed([17F823A1E86E2A03:13F0AC52FACBC522]:0)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:143)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:135)
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:130)
	... 47 more




Build Log:
[...truncated 11326 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.OverseerTest-17F823A1E86E2A03-001/init-core-data-001
   [junit4]   2> 2779 T11 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2913 T11 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 2914 T11 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 2978 T11 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
   [junit4]   2> 2991 T11 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3003 T12 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3203 T11 oasc.ZkTestServer.run start zk server on port:42044
   [junit4]   2> 3594 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3823 T18 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56d96047 name:ZooKeeperConnection Watcher:127.0.0.1:42044 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3824 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3969 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3972 T20 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2464a693 name:ZooKeeperConnection Watcher:127.0.0.1:42044 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3973 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3983 T11 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 4017 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4020 T22 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@115b92b3 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4024 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4025 T11 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 4030 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4033 T24 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bdc8869 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4034 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4290 T11 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 4294 T11 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 4338 T11 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:42044_solr
   [junit4]   2> 4351 T11 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 4355 T11 oasc.Overseer.start Overseer (id=92097285007343619-127.0.0.1:42044_solr-n_0000000000) starting
   [junit4]   2> 4392 T11 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 4398 T11 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 4411 T11 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 4417 T11 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 4422 T11 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 4537 T11 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 4546 T26 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 4547 T11 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 4551 T11 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 4554 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4605 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4608 T28 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@118b05b2 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4609 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4612 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4616 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
   [junit4]   2> 4629 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4632 T30 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e37ee4a name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4633 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4635 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4640 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 4648 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4651 T32 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fec8bf name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4652 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4654 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4657 T25 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 4659 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 4665 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4667 T34 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@472a74c7 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4668 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4671 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4675 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
   [junit4]   2> 4681 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4684 T36 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@785ed376 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4685 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4687 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4691 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
   [junit4]   2> 4696 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4698 T38 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58198636 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4699 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4701 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4705 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
   [junit4]   2> 4711 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4713 T40 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4232ba35 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4714 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4716 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4719 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
   [junit4]   2> 4725 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4727 T42 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@437a50e2 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4728 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4730 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4734 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
   [junit4]   2> 4740 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4743 T44 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d08cee4 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4744 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4746 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4751 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
   [junit4]   2> 4758 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4761 T46 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@302d6b3 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4762 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4765 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4769 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
   [junit4]   2> 4777 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4780 T48 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ba50406 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4781 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4784 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4789 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
   [junit4]   2> 4798 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4801 T50 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3de53889 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4802 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4804 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4809 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
   [junit4]   2> 4821 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4824 T52 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f8451cf name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4825 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4828 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4833 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
   [junit4]   2> 4842 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4844 T54 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14654146 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4845 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4848 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4853 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
   [junit4]   2> 4861 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4868 T56 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c21e866 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4868 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4871 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4875 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
   [junit4]   2> 4881 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4884 T58 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@587b6c50 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4885 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4887 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4892 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
   [junit4]   2> 4900 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4902 T60 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8a6287e name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4903 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4905 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4909 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
   [junit4]   2> 4919 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4922 T62 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65b32b8f name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4923 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4926 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4930 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
   [junit4]   2> 4941 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4943 T64 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@532c81b7 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4944 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4946 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4951 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
   [junit4]   2> 4961 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4964 T66 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77b576f8 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4965 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4967 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4972 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
   [junit4]   2> 4982 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4984 T68 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c2b74c4 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4985 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4988 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4992 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
   [junit4]   2> 5001 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5004 T70 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cb89281 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5005 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5007 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5011 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
   [junit4]   2> 5021 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5023 T72 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4051b1fb name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5024 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5026 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5030 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
   [junit4]   2> 5039 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5042 T74 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f98440e name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5043 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5045 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5050 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node23
   [junit4]   2> 5059 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5061 T76 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b88b5b2 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5062 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5064 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5067 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node24
   [junit4]   2> 5077 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5079 T78 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b700eb2 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5080 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5081 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5084 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node25
   [junit4]   2> 5094 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5096 T80 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c754c30 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5097 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5099 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5102 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node26
   [junit4]   2> 5112 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5114 T82 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c37fd3f name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5114 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5116 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5119 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node27
   [junit4]   2> 5129 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5131 T84 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7172f77a name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5132 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5134 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5136 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node28
   [junit4]   2> 5144 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5146 T86 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3369d7a name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5146 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5148 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5151 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node29
   [junit4]   2> 5158 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5160 T88 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@100a438d name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5160 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5162 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5164 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node30
   [junit4]   2> 5173 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5175 T90 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d5bfc1f name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5176 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5178 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5182 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node31
   [junit4]   2> 5191 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5194 T92 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50adc92c name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5195 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5197 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5200 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node32
   [junit4]   2> 5210 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5212 T94 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d638fac name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5212 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5214 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5216 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node33
   [junit4]   2> 5228 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5230 T96 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12f6ce3e name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5230 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5233 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5236 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node34
   [junit4]   2> 5248 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5250 T98 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d69d193 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5251 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5252 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5255 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node35
   [junit4]   2> 5268 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5270 T100 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fc8ddb5 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5271 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5272 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5275 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node36
   [junit4]   2> 5285 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5287 T102 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@246fa9f9 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5288 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5290 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5294 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node37
   [junit4]   2> 5306 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5308 T104 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71fcf7e2 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5309 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5310 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5313 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node38
   [junit4]   2> 5324 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5326 T106 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e72c39b name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5327 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5329 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5332 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node39
   [junit4]   2> 5346 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5348 T108 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@62c0958a name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5349 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5350 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5353 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node40
   [junit4]   2> 5363 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5407 T110 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64dde202 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5408 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5410 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5412 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node41
   [junit4]   2> 5421 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5423 T112 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7eee0e6f name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5423 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5424 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5427 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node42
   [junit4]   2> 5435 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5438 T114 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@632203aa name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5438 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5440 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5443 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node43
   [junit4]   2> 5461 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5463 T116 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@effa25e name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5464 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5465 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5468 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node44
   [junit4]   2> 5478 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5481 T118 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2804b9c9 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5481 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5482 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5485 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node45
   [junit4]   2> 5495 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5498 T120 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10bc86fc name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5498 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5500 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5502 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node46
   [junit4]   2> 5516 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5518 T122 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c28e5a7 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5518 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5519 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5522 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node47
   [junit4]   2> 5531 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5533 T124 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b0fc9d1 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5534 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5535 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5538 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node48
   [junit4]   2> 5558 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5560 T126 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@421fb7c6 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5560 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5562 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5566 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node49
   [junit4]   2> 5580 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 5582 T128 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15daf565 name:ZooKeeperConnection Watcher:127.0.0.1:42044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5583 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 5585 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 5588 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/node50
   [junit4]   2> 5617 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5627 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5646 T25 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 5648 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 5648 T25 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12, shard13, shard14, shard15, shard16, shard17]
   [junit4]   2> 5725 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 5755 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T102 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T118 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T30 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T42 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T98 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5755 T108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5755 T100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5755 T124 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T126 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T72 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T128 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T66 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T110 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T34 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T36 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T60 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T28 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5758 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T112 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T50 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T74 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T86 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T22 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5757 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T78 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T32 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5756 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 51)
   [junit4]   2> 5770 T25 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 5812 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 5813 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5813 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard13
   [junit4]   2> 5818 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5820 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node0",
   [junit4]   2> 	  "core":"core0",
   [junit4]   2> 	  "core_node_name":"node0",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node0/solr/"}
   [junit4]   2> 5821 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5821 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard7
   [junit4]   2> 5825 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5827 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node3",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node3/solr/"}
   [junit4]   2> 5828 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5829 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard9
   [junit4]   2> 5832 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5834 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node4",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node4",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node4/solr/"}
   [junit4]   2> 5835 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5835 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 5840 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5841 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node5",
   [junit4]   2> 	  "core":"core5",
   [junit4]   2> 	  "core_node_name":"node5",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node5/solr/"}
   [junit4]   2> 5842 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5842 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard6
   [junit4]   2> 5846 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5848 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node6",
   [junit4]   2> 	  "core":"core6",
   [junit4]   2> 	  "core_node_name":"node6",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node6/solr/"}
   [junit4]   2> 5849 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5849 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard10
   [junit4]   2> 5853 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5855 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node8",
   [junit4]   2> 	  "core":"core8",
   [junit4]   2> 	  "core_node_name":"node8",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node8/solr/"}
   [junit4]   2> 5855 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5856 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard5
   [junit4]   2> 5860 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5861 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node7",
   [junit4]   2> 	  "core":"core7",
   [junit4]   2> 	  "core_node_name":"node7",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node7/solr/"}
   [junit4]   2> 5862 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5862 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard17
   [junit4]   2> 5866 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5868 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node9",
   [junit4]   2> 	  "core":"core9",
   [junit4]   2> 	  "core_node_name":"node9",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node9/solr/"}
   [junit4]   2> 5869 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5870 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard16
   [junit4]   2> 5875 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5877 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node10",
   [junit4]   2> 	  "core":"core10",
   [junit4]   2> 	  "core_node_name":"node10",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node10/solr/"}
   [junit4]   2> 5878 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5879 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard8
   [junit4]   2> 5883 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5886 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node11",
   [junit4]   2> 	  "core":"core11",
   [junit4]   2> 	  "core_node_name":"node11",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node11/solr/"}
   [junit4]   2> 5886 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5887 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard11
   [junit4]   2> 5891 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5893 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node13",
   [junit4]   2> 	  "core":"core13",
   [junit4]   2> 	  "core_node_name":"node13",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node13/solr/"}
   [junit4]   2> 5893 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5894 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard14
   [junit4]   2> 5897 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5899 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node14",
   [junit4]   2> 	  "core":"core14",
   [junit4]   2> 	  "core_node_name":"node14",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node14/solr/"}
   [junit4]   2> 5899 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5900 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard12
   [junit4]   2> 5903 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5905 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node12",
   [junit4]   2> 	  "core":"core12",
   [junit4]   2> 	  "core_node_name":"node12",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node12/solr/"}
   [junit4]   2> 5905 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5906 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard4
   [junit4]   2> 5909 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5911 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node15",
   [junit4]   2> 	  "core":"core15",
   [junit4]   2> 	  "core_node_name":"node15",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node15/solr/"}
   [junit4]   2> 5911 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5912 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard15
   [junit4]   2> 5915 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5917 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node16",
   [junit4]   2> 	  "core":"core16",
   [junit4]   2> 	  "core_node_name":"node16",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node16/solr/"}
   [junit4]   2> 5917 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5918 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 5921 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5923 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node17",
   [junit4]   2> 	  "core":"core17",
   [junit4]   2> 	  "core_node_name":"node17",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node17/solr/"}
   [junit4]   2> 5923 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5924 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 5928 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5929 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node18",
   [junit4]   2> 	  "core":"core18",
   [junit4]   2> 	  "core_node_name":"node18",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node18/solr/"}
   [junit4]   2> 5930 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5930 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard13
   [junit4]   2> 5934 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5936 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node19",
   [junit4]   2> 	  "core":"core19",
   [junit4]   2> 	  "core_node_name":"node19",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node19/solr/"}
   [junit4]   2> 5937 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5937 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard7
   [junit4]   2> 5941 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5943 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node20",
   [junit4]   2> 	  "core":"core20",
   [junit4]   2> 	  "core_node_name":"node20",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node20/solr/"}
   [junit4]   2> 5944 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5944 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard9
   [junit4]   2> 5948 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5949 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node21",
   [junit4]   2> 	  "core":"core21",
   [junit4]   2> 	  "core_node_name":"node21",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node21/solr/"}
   [junit4]   2> 5950 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5951 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 5955 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5956 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node22",
   [junit4]   2> 	  "core":"core22",
   [junit4]   2> 	  "core_node_name":"node22",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node22/solr/"}
   [junit4]   2> 5957 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5957 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard6
   [junit4]   2> 5962 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5963 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node23",
   [junit4]   2> 	  "core":"core23",
   [junit4]   2> 	  "core_node_name":"node23",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node23/solr/"}
   [junit4]   2> 5964 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5964 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard10
   [junit4]   2> 5968 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5969 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node24",
   [junit4]   2> 	  "core":"core24",
   [junit4]   2> 	  "core_node_name":"node24",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node24/solr/"}
   [junit4]   2> 5970 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5970 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard5
   [junit4]   2> 5974 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5975 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node27",
   [junit4]   2> 	  "core":"core27",
   [junit4]   2> 	  "core_node_name":"node27",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node27/solr/"}
   [junit4]   2> 5975 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5976 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard17
   [junit4]   2> 5980 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5981 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node28",
   [junit4]   2> 	  "core":"core28",
   [junit4]   2> 	  "core_node_name":"node28",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node28/solr/"}
   [junit4]   2> 5982 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5982 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard16
   [junit4]   2> 5986 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5987 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node25",
   [junit4]   2> 	  "core":"core25",
   [junit4]   2> 	  "core_node_name":"node25",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node25/solr/"}
   [junit4]   2> 5988 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5988 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard8
   [junit4]   2> 5992 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 5994 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node26",
   [junit4]   2> 	  "core":"core26",
   [junit4]   2> 	  "core_node_name":"node26",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node26/solr/"}
   [junit4]   2> 5994 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 5995 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard11
   [junit4]   2> 5998 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6000 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node29",
   [junit4]   2> 	  "core":"core29",
   [junit4]   2> 	  "core_node_name":"node29",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node29/solr/"}
   [junit4]   2> 6000 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 6001 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard14
   [junit4]   2> 6005 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6007 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node30",
   [junit4]   2> 	  "core":"core30",
   [junit4]   2> 	  "core_node_name":"node30",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node30/solr/"}
   [junit4]   2> 6007 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 6008 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard12
   [junit4]   2> 6012 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6014 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node31",
   [junit4]   2> 	  "core":"core31",
   [junit4]   2> 	  "core_node_name":"node31",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node31/solr/"}
   [junit4]   2> 6014 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 6015 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard4
   [junit4]   2> 6018 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6020 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node32",
   [junit4]   2> 	  "core":"core32",
   [junit4]   2> 	  "core_node_name":"node32",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node32/solr/"}
   [junit4]   2> 6020 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 6021 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard15
   [junit4]   2> 6024 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6026 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node35",
   [junit4]   2> 	  "core":"core35",
   [junit4]   2> 	  "core_node_name":"node35",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node35/solr/"}
   [junit4]   2> 6027 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 6028 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 6031 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6033 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node33",
   [junit4]   2> 	  "core":"core33",
   [junit4]   2> 	  "core_node_name":"node33",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node33/solr/"}
   [junit4]   2> 6034 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 6034 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 6037 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6039 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node36",
   [junit4]   2> 	  "core":"core36",
   [junit4]   2> 	  "core_node_name":"node36",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node36/solr/"}
   [junit4]   2> 6039 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 6040 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard13
   [junit4]   2> 6043 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6044 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node34",
   [junit4]   2> 	  "core":"core34",
   [junit4]   2> 	  "core_node_name":"node34",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node34/solr/"}
   [junit4]   2> 6044 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 6045 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard7
   [junit4]   2> 6048 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6049 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node37",
   [junit4]   2> 	  "core":"core37",
   [junit4]   2> 	  "core_node_name":"node37",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node37/solr/"}
   [junit4]   2> 6049 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 6050 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard9
   [junit4]   2> 6053 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6054 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node38",
   [junit4]   2> 	  "core":"core38",
   [junit4]   2> 	  "core_node_name":"node38",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"17",
   [junit4]   2> 	  "base_url":"http://node38/solr/"}
   [junit4]   2> 6055 T25 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=17
   [junit4]   2> 6055 T25 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 6063 T24 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6065 T25 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=17 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node39",
   [junit4]   2> 	  "core":"core39",
   [junit4]   2> 	  

[...truncated too long message...]

erseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44518 T483 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44519 T483 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44520 T484 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 44521 T484 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core6",
   [junit4]   2> 	  "core_node_name":"node6",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 44521 T484 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 44521 T484 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 44523 T483 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44626 T487 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 44626 T481 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 45018 T11 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 45026 T11 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:31482 31482
   [junit4]   2> 45107 T471 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:31482 31482
   [junit4]   2> 45109 T11 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
   [junit4]   2> 45109 T484 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:31482_solr
   [junit4]   2> 45111 T488 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 46525 T487 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 46526 T487 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 46527 T487 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 48114 T11 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.OverseerTest-17F823A1E86E2A03-001
   [junit4]   2> NOTE: test params are: codec=Lucene46, sim=DefaultSimilarity, locale=ms_MY, timezone=Asia/Riyadh
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_60 (64-bit)/cpus=16,threads=1,free=244137712,total=272105472
   [junit4]   2> NOTE: All tests run in this JVM: [OverseerTest]
   [junit4] Completed on J0 in 48.86s, 9 tests, 1 error <<< FAILURES!

[...truncated 1317 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1298: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:922: There were test failures: 418 suites, 1710 tests, 1 error, 50 ignored (8 assumptions)

Total time: 185 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #583
Archived 3 artifacts
Archive block size is 32768
Received 0 blocks and 590583559 bytes
Compression is 0.0%
Took 1 min 33 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 585 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/585/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
Error executing query

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Error executing query
	at __randomizedtesting.SeedInfo.seed([9EF27A84103673F0:1F14F49C676913CC]:0)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:100)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:223)
	at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:165)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrServer(FullSolrCloudDistribCmdsTest.java:415)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:145)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.ParseException: Invalid content type: 
	at org.apache.http.entity.ContentType.parse(ContentType.java:233)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:500)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	... 46 more




Build Log:
[...truncated 11627 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/init-core-data-001
   [junit4]   2> 1432797 T2812 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ksh/
   [junit4]   2> 1432803 T2812 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1432804 T2812 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1432804 T2813 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1432904 T2812 oasc.ZkTestServer.run start zk server on port:65070
   [junit4]   2> 1432905 T2812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1432909 T2819 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74a1d898 name:ZooKeeperConnection Watcher:127.0.0.1:65070 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1432910 T2812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1432910 T2812 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1432913 T2812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1432915 T2821 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@126c9218 name:ZooKeeperConnection Watcher:127.0.0.1:65070/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1432915 T2812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1432915 T2812 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1432918 T2812 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1432919 T2812 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1432921 T2812 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1432923 T2812 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1432924 T2812 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1432927 T2812 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1432927 T2812 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1433030 T2812 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1433031 T2812 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1433033 T2812 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1433034 T2812 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1433036 T2812 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1433036 T2812 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1433038 T2812 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1433039 T2812 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1433041 T2812 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1433042 T2812 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1433044 T2812 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1433044 T2812 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1433047 T2812 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1433047 T2812 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1433049 T2812 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1433050 T2812 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1433052 T2812 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1433052 T2812 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1434098 T2812 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1434103 T2812 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:?????
   [junit4]   2> 1434103 T2812 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1434104 T2812 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1434104 T2812 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-002
   [junit4]   2> 1434104 T2812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-002/'
   [junit4]   2> 1434132 T2812 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-002/solr.xml
   [junit4]   2> 1434185 T2812 oasc.CoreContainer.<init> New CoreContainer 1288743320
   [junit4]   2> 1434185 T2812 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-002/]
   [junit4]   2> 1434186 T2812 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1434186 T2812 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1434187 T2812 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1434187 T2812 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1434187 T2812 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1434187 T2812 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1434187 T2812 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1434188 T2812 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1434188 T2812 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1434188 T2812 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1434189 T2812 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1434189 T2812 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1434189 T2812 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:65070/solr
   [junit4]   2> 1434189 T2812 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1434190 T2812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1434192 T2832 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@172e618e name:ZooKeeperConnection Watcher:127.0.0.1:65070 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1434192 T2812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1434194 T2812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1434195 T2834 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@668e1543 name:ZooKeeperConnection Watcher:127.0.0.1:65070/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1434195 T2812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1434197 T2812 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1434200 T2812 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1434202 T2812 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1434204 T2812 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1434206 T2812 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1434208 T2812 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1434209 T2812 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50176_ksh
   [junit4]   2> 1434210 T2812 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50176_ksh
   [junit4]   2> 1434212 T2812 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1434213 T2812 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1434216 T2812 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50176_ksh
   [junit4]   2> 1434216 T2812 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1434218 T2812 oasc.Overseer.start Overseer (id=92106913571995651-127.0.0.1:50176_ksh-n_0000000000) starting
   [junit4]   2> 1434220 T2812 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1434226 T2836 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1434226 T2812 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1434228 T2812 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1434229 T2812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1434231 T2835 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1434235 T2837 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1434235 T2837 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1434236 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1434236 T2837 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1434237 T2835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1434238 T2835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50176/ksh",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50176_ksh",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1434238 T2835 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1434238 T2835 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1434240 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1434241 T2834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1435237 T2837 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1435238 T2837 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1435238 T2837 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1435239 T2837 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1435239 T2837 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1435240 T2837 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-002/collection1/'
   [junit4]   2> 1435241 T2837 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 1435242 T2837 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 1435242 T2837 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 1435304 T2837 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 1435360 T2837 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1435461 T2837 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1435471 T2837 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1435807 T2837 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1435810 T2837 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1435812 T2837 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1435816 T2837 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1435836 T2837 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1435836 T2837 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1435836 T2837 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-002/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/control/data/
   [junit4]   2> 1435837 T2837 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@355e86a1
   [junit4]   2> 1435839 T2837 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/control/data
   [junit4]   2> 1435839 T2837 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/control/data/index/
   [junit4]   2> 1435840 T2837 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1435842 T2837 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/control/data/index
   [junit4]   2> 1435842 T2837 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@1ce88a16
   [junit4]   2> 1435847 T2837 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(MMapDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/index-MMapDirectory-001 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66c1b92d),segFN=segments_1,generation=1}
   [junit4]   2> 1435847 T2837 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1435853 T2837 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1435853 T2837 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1435853 T2837 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1435854 T2837 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1435854 T2837 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1435854 T2837 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1435855 T2837 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1435855 T2837 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1435855 T2837 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1435856 T2837 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1435856 T2837 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1435856 T2837 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1435857 T2837 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1435857 T2837 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1435857 T2837 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1435858 T2837 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1435858 T2837 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1435877 T2837 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1435883 T2837 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1435883 T2837 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1435884 T2837 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=91.9990234375, floorSegmentMB=1.830078125, forceMergeDeletesPctAllowed=28.581556762841775, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1435886 T2837 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(MMapDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/index-MMapDirectory-001 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66c1b92d),segFN=segments_1,generation=1}
   [junit4]   2> 1435886 T2837 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1435886 T2837 oass.SolrIndexSearcher.<init> Opening Searcher@6461b9f7[collection1] main
   [junit4]   2> 1435887 T2837 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1435888 T2837 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1435888 T2837 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1435888 T2837 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1435889 T2837 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1435889 T2837 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1435889 T2837 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1435890 T2837 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1435890 T2837 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 1435892 T2837 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1435892 T2837 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1435893 T2837 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1435896 T2838 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6461b9f7[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1435898 T2837 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1435899 T2841 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50176/ksh collection:control_collection shard:shard1
   [junit4]   2> 1435899 T2812 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 1435900 T2812 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1435900 T2841 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1435901 T2812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1435902 T2843 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ca9fca1 name:ZooKeeperConnection Watcher:127.0.0.1:65070/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1435903 T2812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1435904 T2812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1435905 T2841 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1435906 T2812 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1435907 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1435907 T2841 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1435907 T2841 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C599 name=collection1 org.apache.solr.core.SolrCore@4d91a184 url=http://127.0.0.1:50176/ksh/collection1 node=127.0.0.1:50176_ksh C599_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:50176/ksh, core=collection1, node_name=127.0.0.1:50176_ksh}
   [junit4]   2> 1435908 T2841 C599 P50176 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50176/ksh/collection1/
   [junit4]   2> 1435908 T2841 C599 P50176 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1435908 T2835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1435908 T2841 C599 P50176 oasc.SyncStrategy.syncToMe http://127.0.0.1:50176/ksh/collection1/ has no replicas
   [junit4]   2> 1435909 T2841 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50176/ksh/collection1/ shard1
   [junit4]   2> 1435909 T2841 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1435910 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1435911 T2843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1435911 T2834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1435914 T2835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1435916 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1436017 T2843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1436017 T2834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1436066 T2841 oasc.ZkController.register We are http://127.0.0.1:50176/ksh/collection1/ and leader is http://127.0.0.1:50176/ksh/collection1/
   [junit4]   2> 1436067 T2841 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50176/ksh
   [junit4]   2> 1436067 T2841 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1436067 T2841 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1436067 T2841 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1436068 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1436068 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1436069 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1436069 T2841 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1436070 T2835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1436070 T2835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50176/ksh",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50176_ksh",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1436072 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1436173 T2843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1436173 T2834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1436989 T2812 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1436990 T2812 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1436992 T2812 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:?????
   [junit4]   2> 1436993 T2812 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1436993 T2812 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1436993 T2812 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-003
   [junit4]   2> 1436993 T2812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-003/'
   [junit4]   2> 1437021 T2812 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-003/solr.xml
   [junit4]   2> 1437073 T2812 oasc.CoreContainer.<init> New CoreContainer 1062233595
   [junit4]   2> 1437073 T2812 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-003/]
   [junit4]   2> 1437074 T2812 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1437074 T2812 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1437074 T2812 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1437075 T2812 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1437075 T2812 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1437075 T2812 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1437075 T2812 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1437076 T2812 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1437076 T2812 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1437076 T2812 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1437077 T2812 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1437077 T2812 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1437077 T2812 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:65070/solr
   [junit4]   2> 1437077 T2812 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1437078 T2812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1437080 T2854 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37576236 name:ZooKeeperConnection Watcher:127.0.0.1:65070 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1437080 T2812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1437082 T2812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1437083 T2856 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a0f98a7 name:ZooKeeperConnection Watcher:127.0.0.1:65070/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1437083 T2812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1437089 T2812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1438092 T2812 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51714_ksh
   [junit4]   2> 1438093 T2812 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51714_ksh
   [junit4]   2> 1438103 T2857 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1438103 T2857 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1438104 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1438104 T2857 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1438104 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1438104 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1438105 T2835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1438106 T2835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51714/ksh",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51714_ksh",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1438106 T2835 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 1438107 T2835 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1438108 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1438109 T2834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1438109 T2843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1438109 T2856 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1439104 T2857 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1439105 T2857 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1439106 T2857 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1439106 T2857 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1439107 T2857 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1439107 T2857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-003/collection1/'
   [junit4]   2> 1439108 T2857 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 1439109 T2857 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 1439110 T2857 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 1439172 T2857 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 1439227 T2857 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1439329 T2857 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1439338 T2857 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1439676 T2857 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1439679 T2857 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1439681 T2857 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1439686 T2857 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1439706 T2857 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1439706 T2857 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1439706 T2857 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-003/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/jetty1/
   [junit4]   2> 1439706 T2857 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@355e86a1
   [junit4]   2> 1439709 T2857 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/jetty1
   [junit4]   2> 1439710 T2857 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/jetty1/index/
   [junit4]   2> 1439710 T2857 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1439712 T2857 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/jetty1/index
   [junit4]   2> 1439713 T2857 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@5d9108b0
   [junit4]   2> 1439717 T2857 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(MMapDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/index-MMapDirectory-002 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5cd18acd),segFN=segments_1,generation=1}
   [junit4]   2> 1439718 T2857 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1439724 T2857 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1439725 T2857 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1439725 T2857 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1439725 T2857 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1439725 T2857 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1439726 T2857 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1439726 T2857 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1439726 T2857 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1439727 T2857 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1439727 T2857 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1439728 T2857 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1439728 T2857 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1439728 T2857 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1439729 T2857 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1439729 T2857 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1439730 T2857 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1439730 T2857 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1439749 T2857 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1439754 T2857 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1439754 T2857 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1439755 T2857 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=91.9990234375, floorSegmentMB=1.830078125, forceMergeDeletesPctAllowed=28.581556762841775, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1439757 T2857 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(MMapDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/index-MMapDirectory-002 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5cd18acd),segFN=segments_1,generation=1}
   [junit4]   2> 1439757 T2857 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1439758 T2857 oass.SolrIndexSearcher.<init> Opening Searcher@17cea8c9[collection1] main
   [junit4]   2> 1439758 T2857 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1439759 T2857 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1439759 T2857 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1439760 T2857 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1439760 T2857 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1439760 T2857 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1439761 T2857 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1439761 T2857 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1439762 T2857 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 1439762 T2857 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1439766 T2858 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@17cea8c9[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1439769 T2857 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1439770 T2861 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51714/ksh collection:collection1 shard:shard1
   [junit4]   2> 1439770 T2812 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 1439770 T2812 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1439771 T2861 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1439775 T2861 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1439777 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439777 T2861 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1439777 T2861 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C600 name=collection1 org.apache.solr.core.SolrCore@8d8ce37 url=http://127.0.0.1:51714/ksh/collection1 node=127.0.0.1:51714_ksh C600_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:51714/ksh, core=collection1, node_name=127.0.0.1:51714_ksh}
   [junit4]   2> 1439777 T2861 C600 P51714 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51714/ksh/collection1/
   [junit4]   2> 1439778 T2861 C600 P51714 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1439778 T2835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1439778 T2861 C600 P51714 oasc.SyncStrategy.syncToMe http://127.0.0.1:51714/ksh/collection1/ has no replicas
   [junit4]   2> 1439778 T2861 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51714/ksh/collection1/ shard1
   [junit4]   2> 1439779 T2861 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1439780 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439781 T2834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1439781 T2843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1439781 T2856 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1439783 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439784 T2835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1439787 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439888 T2856 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1439888 T2843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1439889 T2834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1439936 T2861 oasc.ZkController.register We are http://127.0.0.1:51714/ksh/collection1/ and leader is http://127.0.0.1:51714/ksh/collection1/
   [junit4]   2> 1439937 T2861 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51714/ksh
   [junit4]   2> 1439937 T2861 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1439937 T2861 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1439937 T2861 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1439938 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439939 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439939 T2861 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1439939 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1439940 T2835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1439940 T2835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51714/ksh",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51714_ksh",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1439942 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1440044 T2856 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1440044 T2843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1440044 T2834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1440909 T2812 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1440910 T2812 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1440912 T2812 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:?????
   [junit4]   2> 1440912 T2812 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1440913 T2812 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1440913 T2812 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-004
   [junit4]   2> 1440913 T2812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-004/'
   [junit4]   2> 1440942 T2812 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-004/solr.xml
   [junit4]   2> 1440993 T2812 oasc.CoreContainer.<init> New CoreContainer 1619517851
   [junit4]   2> 1440994 T2812 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-004/]
   [junit4]   2> 1440994 T2812 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1440995 T2812 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1440995 T2812 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1440995 T2812 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1440995 T2812 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1440995 T2812 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1440996 T2812 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1440996 T2812 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1440996 T2812 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1440997 T2812 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1440997 T2812 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1440997 T2812 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1440997 T2812 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:65070/solr
   [junit4]   2> 1440998 T2812 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1440998 T2812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1441000 T2872 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78ecb89f name:ZooKeeperConnection Watcher:127.0.0.1:65070 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1441000 T2812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1441002 T2812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1441003 T2874 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5535fded name:ZooKeeperConnection Watcher:127.0.0.1:65070/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1441003 T2812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1441008 T2812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1442011 T2812 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:19334_ksh
   [junit4]   2> 1442012 T2812 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:19334_ksh
   [junit4]   2> 1442022 T2875 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1442022 T2875 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1442023 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1442023 T2875 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1442023 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1442023 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1442025 T2835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1442025 T2835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:19334/ksh",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:19334_ksh",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1442025 T2835 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1442026 T2835 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 1442027 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1442028 T2856 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1442028 T2843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1442028 T2874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1442028 T2834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1443023 T2875 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1443024 T2875 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1443024 T2875 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1443025 T2875 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1443025 T2875 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1443026 T2875 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-004/collection1/'
   [junit4]   2> 1443028 T2875 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 1443029 T2875 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 1443029 T2875 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 1443091 T2875 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 1443147 T2875 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1443248 T2875 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1443259 T2875 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1443598 T2875 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1443601 T2875 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1443602 T2875 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1443607 T2875 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1443627 T2875 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1443628 T2875 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1443628 T2875 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-004/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/jetty2/
   [junit4]   2> 1443628 T2875 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@355e86a1
   [junit4]   2> 1443631 T2875 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/jetty2
   [junit4]   2> 1443632 T2875 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/jetty2/index/
   [junit4]   2> 1443632 T2875 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1443635 T2875 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-001/jetty2/index
   [junit4]   2> 1443635 T2875 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@1881ec36
   [junit4]   2> 1443679 T2875 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(MMapDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/index-MMapDirectory-003 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@79bad9cb),segFN=segments_1,generation=1}
   [junit4]   2> 1443680 T2875 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1443687 T2875 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1443687 T2875 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1443687 T2875 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1443688 T2875 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1443688 T2875 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1443688 T2875 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1443689 T2875 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1443689 T2875 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1443689 T2875 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1443690 T2875 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1443690 T2875 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1443691 T2875 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1443691 T2875 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1443692 T2875 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1443692 T2875 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1443693 T2875 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1443693 T2875 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1443712 T2875 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1443717 T2875 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1443717 T2875 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1443718 T2875 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=91.9990234375, floorSegmentMB=1.830078125, forceMergeDeletesPctAllowed=28.581556762841775, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1443720 T2875 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(MMapDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/index-MMapDirectory-003 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@79bad9cb),segFN=segments_1,generation=1}
   [junit4]   2> 1443720 T2875 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1443721 T2875 oass.SolrIndexSearcher.<init> Opening Searcher@22f2b1d4[collection1] main
   [junit4]   2> 1443721 T2875 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1443722 T2875 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1443722 T2875 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1443723 T2875 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1443723 T2875 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1443723 T2875 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1443724 T2875 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1443724 T2875 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1443724 T2875 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 1443725 T2875 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1443729 T2876 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@22f2b1d4[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1443732 T2875 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1443733 T2879 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:19334/ksh collection:collection1 shard:shard3
   [junit4]   2> 1443733 T2812 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 1443734 T2812 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1443734 T2879 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 1443740 T2879 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 1443743 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1443743 T2879 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1443743 T2879 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C601 name=collection1 org.apache.solr.core.SolrCore@ab955fd url=http://127.0.0.1:19334/ksh/collection1 node=127.0.0.1:19334_ksh C601_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:19334/ksh, core=collection1, node_name=127.0.0.1:19334_ksh}
   [junit4]   2> 1443743 T2879 C601 P19334 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:19334/ksh/collection1/
   [junit4]   2> 1443744 T2879 C601 P19334 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1443744 T2835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1443744 T2879 C601 P19334 oasc.SyncStrategy.syncToMe http://127.0.0.1:19334/ksh/collection1/ has no replicas
   [junit4]   2> 1443744 T2879 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:19334/ksh/collection1/ shard3
   [junit4]   2> 1443745 T2879 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 1443748 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1443749 T2856 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1443749 T2874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1443749 T2843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1443749 T2834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1443753 T2835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1443756 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1443858 T2843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1443858 T2874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1443858 T2834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1443858 T2856 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1443904 T2879 oasc.ZkController.register We are http://127.0.0.1:19334/ksh/collection1/ and leader is http://127.0.0.1:19334/ksh/collection1/
   [junit4]   2> 1443905 T2879 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:19334/ksh
   [junit4]   2> 1443905 T2879 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1443905 T2879 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1443905 T2879 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1443906 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1443907 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1443907 T2879 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1443907 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1443908 T2835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1443909 T2835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:19334/ksh",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:19334_ksh",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1443910 T2834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1444012 T2874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1444012 T2834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1444012 T2843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1444012 T2856 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1444920 T2812 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1444921 T2812 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1444924 T2812 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:?????
   [junit4]   2> 1444924 T2812 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1444924 T2812 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1444925 T2812 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-005
   [junit4]   2> 1444925 T2812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-005/'
   [junit4]   2> 1444966 T2812 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-005/solr.xml
   [junit4]   2> 1445036 T2812 oasc.CoreContainer.<init> New CoreContainer 456314248
   [junit4]   2> 1445037 T2812 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.FullSolrCloudDistribCmdsTest-9EF27A84103673F0-001/tempDir-005/]
   [junit4]   2> 1445038 T2812 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1445038 T2812 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1445038 T2812 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1445038 T2812 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1445038 T2812 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1445039 T2812 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1445039 T2812 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1445039 T2812 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1445039 T2812 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1445040 T2812 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1445040 T2812 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1445040 T2812 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1445041 T2812 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:65070/solr
   [junit4]   2> 1445041 T2812 oasc.ZkControll

[...truncated too long message...]

, 3child0_s=PostingsFormat(name=Asserting), 49parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 7parent_f1_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 53parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 3child2_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 22parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 6grand0_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 8grand1_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 7grand3_s=PostingsFormat(name=Lucene41WithOrds), 54parent_f1_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 11grand1_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 17parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 41parent_f1_s=PostingsFormat(name=Asserting), 56parent_f1_s=PostingsFormat(name=Asserting), 8grand6_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 25parent_f1_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 1parent_f1_s=PostingsFormat(name=Asserting), 43parent_f2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 10grand9_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 31parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 9grand8_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 15parent_f2_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 10grand8_s=PostingsFormat(name=Asserting), 42parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 57parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 19parent_f2_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 1grand0_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 9grand3_s=PostingsFormat(name=Lucene41WithOrds), 5child1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 7grand1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 4parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 11grand7_s=PostingsFormat(name=Asserting), 16parent_f1_s=PostingsFormat(name=Asserting), 27parent_f1_s=PostingsFormat(name=Asserting), 5child3_s=PostingsFormat(name=Lucene41WithOrds), _version_=PostingsFormat(name=Lucene41WithOrds), 11grand4_s=PostingsFormat(name=Lucene41WithOrds), 23parent_f1_s=PostingsFormat(name=Asserting), 4grand2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), rnd_b=PostingsFormat(name=Asserting), 18parent_f1_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 47parent_f2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 27parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 37parent_f2_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 7grand4_s=PostingsFormat(name=Asserting), 24parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 56parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 10grand5_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 8grand7_s=PostingsFormat(name=Asserting), 3child1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 6parent_f2_s=PostingsFormat(name=Asserting), 11grand8_s=PostingsFormat(name=Lucene41WithOrds), 58parent_f1_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 11grand5_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 10grand1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 11grand3_s=PostingsFormat(name=Asserting), 10grand7_s=PostingsFormat(name=Lucene41WithOrds), 32parent_f1_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 6grand4_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 37parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 15parent_f1_s=PostingsFormat(name=Lucene41WithOrds), a_t=PostingsFormat(name=Lucene41VarGapFixedInterval), 45parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 7parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 5parent_f2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 46parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 9grand6_s=PostingsFormat(name=Asserting), 5grand2_s=PostingsFormat(name=Asserting), 5parent_f1_s=PostingsFormat(name=Asserting), 6grand2_s=PostingsFormat(name=Lucene41WithOrds), 34parent_f1_s=PostingsFormat(name=Asserting), 10parent_f1_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 40parent_f2_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 26parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 17parent_f2_s=PostingsFormat(name=Asserting), 38parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 4child2_s=PostingsFormat(name=Lucene41WithOrds), 6child2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 5grand1_s=PostingsFormat(name=Lucene41WithOrds), 0parent_f2_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 12parent_f1_s=PostingsFormat(name=Asserting), 6grand5_s=PostingsFormat(name=Asserting), 6child5_s=PostingsFormat(name=Asserting), 13parent_f2_s=PostingsFormat(name=Asserting), 2grand0_s=PostingsFormat(name=Lucene41WithOrds), 5child0_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 23parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 6child1_s=PostingsFormat(name=Asserting), 20parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 11parent_f2_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 10grand0_s=PostingsFormat(name=Asserting), 7grand0_s=PostingsFormat(name=Asserting), 28parent_f2_s=PostingsFormat(name=Asserting), 0parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 21parent_f1_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 2child0_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 4child1_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 14parent_f2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 8grand2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 49parent_f1_s=PostingsFormat(name=Asserting), 9grand4_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 5child2_s=PostingsFormat(name=Asserting), 4grand0_s=PostingsFormat(name=Lucene41WithOrds), 3parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 3parent_f1_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 7grand5_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 26parent_f2_s=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 2parent_f2_s=PostingsFormat(name=Asserting)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=th_TH_TH_#u-nu-thai, timezone=America/Montserrat
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_60 (64-bit)/cpus=16,threads=1,free=81332488,total=460849152
   [junit4]   2> NOTE: All tests run in this JVM: [TestRandomFaceting, SliceStateTest, TestBadConfig, TestCopyFieldCollectionResource, TestStressUserVersions, TestPHPSerializedResponseWriter, ChaosMonkeySafeLeaderTest, CSVRequestHandlerTest, RankQueryTest, TestCursorMarkWithoutUniqueKey, TestCollationFieldDocValues, TestCoreContainer, TestFreeTextSuggestions, DeleteReplicaTest, SolrInfoMBeanTest, TestAtomicUpdateErrorCases, TestDistributedGrouping, TestDynamicFieldCollectionResource, TestSchemaNameResource, AlternateDirectoryTest, AddBlockUpdateTest, TestNoOpRegenerator, ShardSplitTest, TestSystemIdResolver, TestMultiCoreConfBootstrap, TestPerFieldSimilarity, TestShortCircuitedRequests, IndexSchemaRuntimeFieldTest, DOMUtilTest, SSLMigrationTest, TestLFUCache, DefaultValueUpdateProcessorTest, DateRangeFieldTest, SolrCoreTest, ConvertedLegacyTest, TestJmxMonitoredMap, ShardRoutingCustomTest, SchemaVersionSpecificBehaviorTest, TestQueryTypes, TestUpdate, DirectUpdateHandlerOptimizeTest, OverseerRolesTest, TestIndexSearcher, AsyncMigrateRouteKeyTest, TestPostingsSolrHighlighter, FullSolrCloudDistribCmdsTest]
   [junit4] Completed on J1 in 326.95s, 1 test, 1 error <<< FAILURES!

[...truncated 955 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1298: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:922: There were test failures: 418 suites, 1710 tests, 1 error, 50 ignored (8 assumptions)

Total time: 236 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #583
Archived 3 artifacts
Archive block size is 32768
Received 0 blocks and 463566579 bytes
Compression is 0.0%
Took 1 min 15 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure